DSFA-SwinNet: A Multi-Scale Attention Fusion Network for Photovoltaic Areas Detection

Abstract

Precise statistics on the spatial distribution of photovoltaics (PV) are essential for advancing the PV industry, and integrating remote sensing with artificial intelligence technologies offers a robust solution for accurate identification. Currently, numerous studies focus on the detection of single-type PV installations through aerial or satellite imagery. However, due to the variability in scale and shape of PV installations in complex environments, the detection results often fail to capture detailed information and struggle to scale for multi-scale PV systems. To tackle these challenges, a detection method known as Dynamic Spatial-Frequency Attention SwinNet (DSFA-SwinNet) for multi-scale PV areas is proposed. First, this study proposes the Dynamic Spatial-Frequency Attention (DSFA) mechanism, the Pyramid Attention Refinement (PAR) bottleneck structure, and optimizes the feature propagation method to achieve dynamic decoupling of the spatial and frequency domains in multi-scale representation learning. Secondly, a hybrid loss function has been developed with weights optimized employing the Bayesian Optimization algorithm to provide a strategic method for parameter tuning in similar research. Lastly, the fixed window size of Swin-Transformer is dynamically adjusted to enhance computational efficiency and maintain accuracy. The results on two PV datasets demonstrate that DSFA-SwinNet significantly enhances detection accuracy and scalability for multi-scale PV areas.

1. Introduction

As global demand for renewable energy rises, solar photovoltaic (PV) technology, a pivotal form of renewable energy, has experienced remarkable growth worldwide. The National Energy Administration (NEA) of China reports in its latest construction update for 2023 that newly installed PV capacity of China hits a record 216.88 GW by December 2023, representing a 148% increase compared to the previous year [1]. These achievements herald the swift progression of the PV industry in the renewable energy sector and establish a robust foundation for the sustainable utilization of clean energy in the future.

However, the accelerated growth of photovoltaic technology, while a key catalyst in the evolution of the clean and renewable energy sector, has unintentionally given rise to a variety of issues, including land degradation, environmental pollution, and the encroachment on arable land [2,3,4]. Driven by the goal of enhancing the environmental compatibility of PV technologies and reducing their potential detrimental effects on socio-ecological systems, the imperative for precise detection of PV areas has escalated. Currently, this process is beset by formidable challenges:

• The geographical distribution of PV installations exhibits significant unevenness, indicating a lack of real-time coordination strategies. This results in an inability to integrate fine-grained data effectively, which constrains the maintenance of the facilities and the assessment of their eco-efficiency;
• The morphological complexity and textural diversity of PV areas pose a significant challenge to accurately identify their detailed features;
• Existing methods focus on single spatial forms, overlooking the diverse scales of PV installations.

Conventional visual detection methods for PV areas frequently demand substantial resources, struggle to meet the need for rapid response within a short period, and face challenges in accurately detecting extensive areas. Consequently, the automatic and precise segmentation of PV areas through high-resolution remote sensing imagery (HRSI) is particularly urgent and has garnered broad interest from both industrial and academic researchers [5,6,7]. Current methods for large-scale PV areas detection predominantly rely on machine learning [8,9,10] and deep learning [11,12].

Machine learning methods primarily focus on the extraction of morphological features or artificially designed features [13]. These morphological features include shape, scale, spectral, and texture, which can be obtained through morphological operations such as binarization, expansion, and closed operations, etc. [14,15]. Following the feature extraction process, researchers construct shallow models such as RF, Support Vector Machine (SVM), and Extreme Gradient Boosting (XGBoost) using manually designed features [16]. Malof et al. are the first to deploy automated remote sensing to assess distributed solar PV [17]. Chen et al. use raw spectral features, photovoltaic extraction indices, and topographic features as XGBoost classification features to extract time-series data of large-scale photovoltaic power plants for the first time from medium-resolution imagery [18]. Li et al. design the SolarFinder system, which combines the advantages of SVM and CNN with a linear regression method to achieve better true positives and true negatives in the PV array detection task [19]. Despite the success of these methods, manual feature design limits the generalization of these methods due to challenges in capturing imaging conditions and PV material properties [20]. Advancing deep learning in remote sensing provides adaptive feature extraction for improved large-scale PV areas detection.

Current deep learning detection methods for PV areas can be broadly categorized into four types, namely CNN [21,22], FCN [23,24], models derived from FCN (e.g., DeepLabV3+, SegNet, Unet, FPN, etc.) [25,26,27], and Visual Transformer [28].

CNN-based methods provide a technical foundation for the application of deep learning in the field of PV detection. Yu et al. develop DeepSolar, a CNN-based deep learning framework, and successfully create a high-fidelity solar deployment database covering the contiguous U.S. [29]. Castello et al. propose a CNN-based pixel-level image segmentation method for obtaining the location and size information of rooftop PV panels [30]. Following the emergence of FCN, it adopts convolutional layers instead of fully connected layers and generates feature maps with the same size as the input image through upsampling, supporting inputs of arbitrary size and enabling end-to-end pixel-level prediction. Yuan et al. are pioneers in proposing FCN model for the identification of distributed PV power plants [31]. Based on this, Sizkouhi et al. use the Mask-RCNN architecture to enhance the detection accuracy of PV plant boundaries [32]. While FCN improves the recognition accuracy of PV areas, it faces issues such as the loss of details in the feature maps and the inability of a fixed receptive field to capture the global contextual relationships when dealing with small objects in complex scenes. To curtail detail loss and broaden the receptive field, researchers have turned to FCN-derived semantic segmentation models including DeepLabV3+ [33], SegNet [34], Unet [35], etc. Though the previous studies have provided insights into detecting PV areas, the limited receptive field of convolutional methods, constrained by their kernel size, impacts the capture of large-scale feature relationships [36,37,38]. In 2021, Dosovitskiy et al. initially propose Visual Transformer (ViT) model [39], which enables Transformer to process image data. Subsequently, a range of remote sensing methods integrating CNN and Transformer have emerged [40,41]. Chen et al. use a Trans-UNet-based architecture to analyze multi-year satellite data for PV variations in China [42]. Guo et al. propose the TransPV model by combining Unet and Visual Transformer, which further enhances the ability of the model to comprehend the global context [43].

On the other hand, to further improve detection performance and overcome the challenges posed by the heterogeneous textures and color features of PV areas, related studies have proposed various deep learning model enhancement strategies, including perceptual enhancement, positive and negative sample balancing, and multi-scale feature optimization.

In the realm of perceptual enhancement, beyond the aforementioned explorations ranging from CNN to FCN-derived semantic segmentation models and extending to Visual Transformer approaches, several studies have explored how attention mechanisms influence the extraction of PV features. Hou et al. develop an Expectation Maximization Attention (EMA) module that utilizes clustering to enhance spatial feature capture [44]. Zhu et al. incorporate a Dual-Attention Module (DAM) to dynamically merge local features with their global dependencies [45].

To manage the balance between positive and negative samples, many studies have adjusted the loss function. Guo et al. introduce Focal loss for mining hard samples [43], while Zhu et al. add IoU loss to address the imbalance between foreground and background in positive and negative samples [45]. However, most hybrid loss functions are constructed without discussing the weight relationship of each loss, and their default weights may not be truly applicable to PV features.

Regarding multi-scale feature optimization, numerous studies have leveraged the Atrous Spatial Pyramid Pooling (ASPP) concept [46]. Tan et al. enhance the ASPP structure for the horizontal features of PV arrays, focusing on processing the contextual features output from the encoder [47]. Moreover, other researchers have explored ways to augment the multi-scale feature extraction capabilities of the model at the data and model training levels. Kleebauer et al. propose hyper-parameter tuning of the PV detection model to adapt it to images of various resolutions [48].

Despite the rich technical support provided by the aforementioned studies for PV areas detection, existing methods still exhibit limitations in global feature extraction, multi-scale feature modeling, and computational efficiency.

• As depicted in Figure 1, there is a significant scale difference between PV arrays and panels in images. While Swin-Transformer [49] mitigates computational complexity with windowing and hierarchies, its fixed window size limits the capture of multi-scale features, leading to an absence of internal multi-scale information within the model [50,51,52];
• In feature learning, existing enhancement strategies do not sufficiently account for the specific characteristics of different PV areas, making it difficult to dynamically adapt to the multi-scale and multi-textural nature of PV features. Furthermore, they lack interpretability regarding hyperparameter tuning;
• Challenges arise as higher image resolutions lead to longer feature sequences, slowing down attention computations and reducing efficiency.

To address the issues mentioned above, this study proposes a multi-scale attention fusion network called Dynamic Spatial-Frequency Attention Swin-Transformer Network (DSFA-SwinNet). The contributions of this study are summarized as follows:

Firstly, to tackle the issue of high intra-class variation and inter-class resemblance in PV areas in HRSI, this study proposes a dynamic window size adjustment mechanism (DWA). This mechanism dynamically adjusts the feature window size based on the long-term token dependency mining mechanism of Swin-Transformer, and effectively captures both the local and global dependencies of multi-scale features in HRSI.

Secondly, this study introduces the Dynamic Spatial-Frequency Attention (DSFA) mechanism and the multi-scale feature refinement bottleneck structure Pyramid Attention Refinement (PAR). These methods significantly bolster the performance of the model by dynamically decoupling the spatial and frequency domains within multi-scale representation learning.

Lastly, this study puts forward the refined skip connection strategy and the depth-supervision-based Multi-Level Upsampling Head (MLUH) module, aiming to augment the representation learning capabilities for PV areas by refining the feature propagation mechanism.

The structure of this study is organized as follows: Section 2 details the datasets, explains the theoretical architecture, and outlines the step-by-step development process of DSFA-SwinNet; Section 3 details the datasets and evaluation criteria employed to assess the performance of the model and presents the experimental results; Section 4 addresses the limitations of the model and proposes potential future enhancements; Section 5 outlines the conclusions.

2. Materials and Methods

2.1. Materials

2.1.1. BDAPPV Dataset

This study employs the Google subset of the BDAPPV dataset [53], as shown in Figure 2. This subset contains 13,303 images from Google Earth with a spatial resolution of 0.1 and a resolution of 400 × 400 pixels.

2.1.2. Jiangsu PV Dataset

To enhance PV sample diversity, this study employs the PV03 subset of a multi-resolution PV Dataset from Jiangsu Province, China [54], containing 2308 satellite and aerial images, as shown in Figure 3. With a spatial resolution of 0.3 m and image size of 1024 × 1024 pixels, the PV03 subset includes both rooftop and ground-based PV systems in areas like shrubland, grassland, cropland, saline-alkali land, and water surfaces.

2.2. DSFA-SwinNet

As shown in Figure 4, the architecture of DSFA-SwinNet is outlined in this section, encompassing the encoder-decoder U-structure architecture, the DSFA mechanism, the PAR bottleneck structure, the refined skip connection strategy, and the hybrid loss function for training.

2.2.1. Swin-Transformer Based U-Model Architecture

DSFA-SwinNet employs Swin-Transformer as the encoder backbone and refers to the ideas of Ren et al. [55] to construct a dynamic window size adjustment mechanism that applies different windows to different channels to achieve multi-scale feature extraction at different levels.

The structure of the dynamic window size adjustment mechanism is depicted in Figure 5. The mechanism constructs a pyramid structure that divides the input feature maps and computes window attention in a distributed manner by utilizing multiple parallel branches, each equipped with distinct window sizes along the channel dimension. This algorithm effectively mitigates the constraints imposed by a fixed window size on the extraction of representational information in traditional vision Transformers, without incurring a substantial increase in computational load. Moreover, it notably enhances the performance of DSFA-SwinNet in the multi-scale representation learning of PV areas features.

The initial size of the input tensor of DSFA-SwinNet is noted as (H, W, C), and multiple patches of size (${\displaystyle \frac{H}{4}}$, ${\displaystyle \frac{W}{4}}$, 96) are generated after window splitting. Each patch is considered as a token and fed into the downsampling layer of Swin-Transformer to deduce feature representations. DSFA-SwinNet utilizes each stage of Swin-Transformer as a layer in Unet encoder, the input tensor with shape ($H_{cur}$, $W_{cur}$, $C_{cur}$) will be downsampled to (${\displaystyle \frac{H_{cur}}{2}}$, ${\displaystyle \frac{W_{cur}}{2}}$, 2$C_{cur}$) after traversal through each layer, with the final encoder outputting feature maps with the shape (${\displaystyle \frac{H}{32}}$, ${\displaystyle \frac{W}{32}}$, 8C). Furthermore, to ensure that DSFA-SwinNet can acquire multi-scale features, the output of each stage is stored in a cache, which is then leveraged in the decoder to execute the refined skip connection strategy.

In the decoder, the DecoderBlock is employed to progressively restore the spatial dimensions of the feature maps through upsampling. Each DecoderBlock is composed of one upsampling layer followed by two convolutional layers. After processing by DecoderBlock module, the input feature maps with shape (${\displaystyle \frac{H_{cur}}{2}}$, ${\displaystyle \frac{W_{cur}}{2}}$, 2$C_{cur}$) will be upsampled to ($H_{cur}$, $W_{cur}$, $C_{cur}$). Within the refined skip connection strategy, the output feature maps are integrated with the skip-connection outcomes at matching resolutions. These are then subjected to the DSFA mechanism to augment the multi-scale information embedded within the feature maps. The output feature maps from both the bottleneck structure and each decoder layer are cached, as indicated by the box called Multi-Level Upsampling Head in Figure 4. These feature maps are converted to feature maps of shape ($H_{cur}$, $W_{cur}$, Class) by 3 × 3 convolutional layers in the MLUH module, and then upsampled to the original image resolution. During training, the deeply supervised approach is adopted, employing output feature maps from the bottleneck structure and the last three decoder layers for auxiliary loss and the top layer for the primary prediction task. During inference, only output feature maps from the top decoder layer are utilized for the main loss. With this multi-scale and deep supervised learning framework, DSFA-SwinNet is more proficient in capturing feature information at various scales.

2.2.2. Dynamic Spatial-Frequency Attention

As the network depth increases, original feature maps evolve from capturing low-level to high-level features, potentially leading to the loss of details. To overcome this issue, this study has engineered the DSFA mechanism to achieve fine-grained PV areas detection by dynamically decoupling multi-scale representation learning in the spatial and frequency domains. The architecture of the DSFA mechanism is delineated in Figure 6. Within the DSFA mechanism, the input feature maps $X_i$, shaped as ($C_{cur}$, $H_{cur}$, $W_{cur}$), are initially directed into the concurrent pathways of channel-space attention and channel-frequency attention.

In the channel-space attention path, features from the $H_{cur}$ and $W_{cur}$ dimensions are independently reconstructed, the processing results of the maximum pooling and average pooling operations are concatenated, and spatial attention weight maps are dynamically generated through the 3 × 3 convolutional layer combined with the Sigmoid function. These maps are then multiplied and summed with the reconstructed features to learn the importance of the feature maps at different spatial locations. Finally, the feature maps are reshaped into the shape of ($C_{cur}$, $H_{cur}$, $W_{cur}$) to obtain the outputs $X_{csh}$ and $X_{csw}$ for this path.

In the channel-frequency attention path, the Discrete Cosine Transform (DCT) is harnessed to craft a frequency attention mechanism. Figure 7 illustrates the application process of DCT. According to the experimental findings of Qin et al. [56], a predefined set of 16-band frequency position indices is established to direct the creation of the DCT filter, as represented by ${freq}_x$ and ${freq}_y$ in Figure 7. The input feature maps are split along the channel dimension into 16 parts. DCT is applied according to each frequency index, resulting in a corresponding set of DCT filters. These filters are sized to match the spatial dimensions of the input features, ensuring comprehensive representation of each feature dimension across different frequency bands. The input features are then translated into the frequency domain through an element-wise multiplication with the DCT filters. The mapping relationship from the spatial domain to the frequency domain is as follows:

$$F_i[u_i,v_i]=\lambda \left(u_i\right)\lambda (v_i)\sum _{x=0}^{H-1}\sum _{y=0}^{W-1}{T}_i[x,y]\cos [{\displaystyle \frac{\pi }{H}}(x+{\displaystyle \frac{1}{2}})u_i]\cos [{\displaystyle \frac{\pi }{W}}(y+{\displaystyle \frac{1}{2}})v_i]$$

(1)

For the ith tensor slice $X_i$ (i = 0, 1, 2, …, 15), $F_i[u_i,v_i]$ represents the 2D frequency space coefficients, and $T_i[x,y]$ represents the tensor values in the spatial domain. H and W are the resolutions of the input tensor, while $u_i$ and $v_i$ are the frequency coordinates, defined as follows:

$$u_i={freq}_x^i$$

(2)

$$v_i={freq}_y^i$$

(3)

$\lambda \left(u_i\right)$ and $\lambda \left(v_i\right)$ are the regularization coefficients, and their definitions are shown in Equations (4) and (5).

$$\lambda \left(u_i\right)=\left\{\begin{array}{c}\sqrt{{\displaystyle \frac{1}{H}}},u_i=0\\ \sqrt{{\displaystyle \frac{2}{H}}},\mathrm{else}\end{array}\right.$$

(4)

$$\lambda \left(v_i\right)=\left\{\begin{array}{c}\sqrt{{\displaystyle \frac{1}{W}}},v_i=0\\ \sqrt{{\displaystyle \frac{2}{W}}},\mathrm{else}\end{array}\right.$$

(5)

Subsequently, the 3 × 3 convolutional layer combined with the Sigmoid function dynamically produces the frequency-attention weight maps. They are then employed to combine and aggregate with the reconstructed feature maps, culminating in the path outputs $X_{cf}$.

The process is advanced by utilizing weight fusion to integrate the outputs from both attention paths. $X_{csh}$ and $X_{csw}$ are concatenated as $X_{cs}$, and then the Sigmoid function is employed to adjust features of $X_{cs}$ and $X_{cf}$ to derive the new weights of attention $\mu$ after the space-frequency interaction. The recalibrated weights $\mu$ are then split into weights ${\mu }_1$ and ${\mu }_2$ corresponding to the channel-space and channel-frequency, and a dot-multiplication operation is performed with the feature maps $X_{cs}$, $X_{cf}$ by element, respectively. The resultant weighted feature maps $X_i^{\prime }$ are produced. The feature maps that have been subjected to the DSFA mechanism exhibit an enhanced representation of multi-scale information. This multidimensional feature fusion capability allows DSFA-SwinNet to concentrate on and capitalize on key features more effectively, while filtering out irrelevant background noise.

2.2.3. Pyramid Attention Refinement

Previous research [57] frequently connects the output from the full Swin-Transformer encoder directly to the decoder, neglecting the integration of intermediate features. This may cause disruptions in the continuous alignment of PV array features or lead to smaller PV panels being overlooked in the imagery.

To bolster the capacity of the model to assimilate the intrinsic details within feature maps, as depicted by the box called Pyramid Attention Refinement in Figure 4, this study incorporates the PAR bottleneck structure, which draws inspiration from the ASPP module. The feature fusion technology of the PAR bottleneck structure is elucidated in Figure 8, showcasing the integration of the output feature maps from the final two layers of the encoder (${\displaystyle \frac{H}{16}}$×${\displaystyle \frac{W}{16}}$, ${\displaystyle \frac{H}{32}}$×${\displaystyle \frac{W}{32}}$) via upsampling and concatenation. Pyramidal parallel pathways are established by employing 1 × 1 convolutional layers intertwined with 3 × 3 convolutional layers possessing diverse dilation rates [2,4,8], thereby encompassing a spectrum of receptive field sizes to capture multi-scale contextual insights of the input features. Thereafter, the conjoined features are steered into the DSFA mechanism designed to distill the intrinsic spatial and frequency bi-dimensional information. The PAR bottleneck structure mitigates the erosion of profound feature information and guarantees the accuracy of PV area extraction throughout the sampling phase on DSFA-SwinNet.

2.2.4. Refined Skip Connection Strategy

As delineated by the box called Refined Skip Connection Strategy in Figure 4, three intermediary units are introduced into the network architecture, designed to nurture the interplay and conveyance of information between multi-scale features via dense interconnections, including not only the traditional skip connections but also the integration of cross-layer features.

Within the encoder, the output feature maps (${\displaystyle \frac{H}{4}}$×${\displaystyle \frac{W}{4}}$, ${\displaystyle \frac{H}{8}}$×${\displaystyle \frac{W}{8}}$, ${\displaystyle \frac{H}{16}}$×${\displaystyle \frac{W}{16}}$) of each downsampled layer except the last layer of the encoder are temporarily stored, which are denoted as $X_{0,0}$, $X_{1,0}$, and $X_{2,0}$, respectively, and the intermediate feature maps, $X_{0,1}$,$X_{1,1}$, and $X_{0,2}$, are acquired through the dense connectivity paths of the intermediate units. For the feature maps emanating from the deeper layers, an initial upsampling operation is requisite, followed by a serial connection of feature maps of equivalent resolution. The connectivity (6)–(8) for the intermediate feature maps are as follows:

$${X_{0,1}=X}_{0,0}\oplus \mathrm{U}\mathrm{p}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\left(X_{1,0}\right)$$

(6)

$${X_{1,1}=X}_{1,0}\oplus \mathrm{U}\mathrm{p}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\left(X_{2,0}\right)$$

(7)

$${X_{0,2}=X}_{0,1}\oplus \mathrm{U}\mathrm{p}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\left(X_{1,1}\right)$$

(8)

The output feature maps processed by the PAR bottleneck structure are denoted as $X_{3,0}^{\prime }$, and the skip-connections (9)–(11) of the decoder are as follows:

$${X_{2,1}=X}_{2,0}\oplus X_{3,0}^{\prime }$$

(9)

$$X_{1,2}=X_{1,1}\oplus X_{1,0}\oplus X_{2,1}^{\prime }$$

(10)

$$X_{0,3}=X_{0,0}\oplus X_{0,1}\oplus X_{0,2}\oplus X_{1,2}^{\prime }$$

(11)

where $X_{2,1}^{\prime }$, $X_{1,2}^{\prime }$ are the feature maps of $X_{2,1}$, $X_{1,2}$, respectively after processing by the sampling module on the decoder.

2.2.5. Loss Function

In DSFA-SwinNet, the PV area detection task is framed as a binary classification problem, with the goal of categorizing each pixel in the image as either “PV area” or “non-PV area”. However, this task is characterized by a significant imbalance in the proportion of positive to negative samples within the labeling.

Table 1. Ratio of PV area pixels to non-PV area pixels in the experimental dataset.

Dataset	Ratio (PV:Not PV)
BDAPPV (Google)	1:23.09
BDAPPV (IGN)	1:75.43
PV01	1:1.32
PV03	1:0.89

presents the sample ratios for the BDAPPV Rooftop PV dataset and the Jiangsu PV dataset.

To tackle the issues arising from imbalanced samples, this study comprehensively considers three different loss functions: the Weighted Binary Cross-Entropy loss (WBCE), the Dice loss, and the Lovasz-Softmax loss [58]. The optimal weights for these functions are determined through theoretical analysis and experiments.

The function equations of WBCE loss are shown in (12)–(14):

$$L_{WBCE}=-{\displaystyle \frac{1}{N}}\sum _{n=1}^N\left({\omega }_0{y}_n\log {\widehat{y}}_n+{\omega }_1\left(1-y_n\right)\log \left(1-{\widehat{y}}_n\right)\right)$$

(12)

where $y_n$ is the actual value of the nth pixel, ${\widehat{y}}_n$ is the predicted value of the nth pixel, N is the total number of pixels, ${\omega }_0$ and ${\omega }_1$ are the weights of the positive and negative classes, respectively, and the calculation expressions are as follows:

$${\omega }_{0 }={\displaystyle \frac{N}{y_n}}$$

(13)

$${\omega }_{1 }={\displaystyle \frac{N}{N-y_n}}$$

(14)

The function equation of Dice loss is shown in (15):

$$L_{Dice}=1-{\displaystyle \frac{2\left|y\cap \widehat{y}\right|}{\left|y\right|+\left|\widehat{y}\right|}}$$

(15)

where y represents the actual value of the nth pixel and $\widehat{y}$ denotes the predicted value of the nth pixel.

For the feature map $S_i$ (i = 1, 2, 3, 4, 5) produced by the bottleneck structure and the decoder, the loss function is expressed as:

$${Loss}_i={\alpha L}_{WBCE}+\beta L_{Dice}+\gamma L_{LS}$$

(16)

where α, β, and γ represent the weight parameters for $L_{WBCE}$, $L_{Dice}$, and $L_{Ls}$, respectively.

The overall loss function for DSFA-SwinNet training is as follows:

$$Loss=\sum {W_i\times Loss}_i$$

(17)

where $W_i$ (i = 1, 2, 3, 4, 5) corresponds to $f_i$ (i = 1, 2, 3, 4, 5) in Figure 4, respectively.

3. Results

3.1. Experimental Setup

All experiments are conducted on a server equipped with 32 GB of RAM and an NVIDIA Quadro P6000 GPU with 16 GB of memory.

3.2. Evaluation Metrics

This study utilizes Recall, Precision, F1 Score, and IoU as evaluation metrics, which are defined as shown in Equations (18) to (21) below.

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(18)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(19)

$$F1={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}={\displaystyle \frac{2TP}{2TP+FN+FP}}$$

(20)

$$IoU={\displaystyle \frac{TP}{TP+FP+FN}}$$

(21)

where TP represents the proportion of PV area pixels that are accurately detected, TN indicates the proportion of background pixels that are accurately classified, FP corresponds to the proportion of background pixels mistakenly identified as PV area, and FN indicates the proportion of PV area pixels misclassified as background.

3.3. Preprocessing and Parameterization

3.3.1. Data Preprocessing

The experimental models accept inputs of sizes that are powers of 2. Considering the original data sizes of the two datasets, a 256 × 256 resolution is chosen as the image input size. The HRSI data preprocessing involves standardizing the PV area labels, cropping the images, and filtering out samples with background pixel counts below 10% or above 90%. For the Google subset, images are directly cropped at the center point. For the PV03 subset, 1024 × 1024 images are cropped into 16 non-overlapping patches. This study applies horizontal and vertical flips for data augmentation and divides the datasets into training, validation, and testing sets in the ratios of 60%, 20%, and 20%, respectively. Detailed information is given in

Table 2. Division of the datasets.

Dataset	Input Size	Original Data	Processed Data	Training Set	Validation Set	Test Set
BDAPPV (Google)	256 × 256	13,303	13,302	7981	2660	2661
PV03	256 × 256	2308	11,593	6955	2318	2320

.

3.3.2. Hyperparameter Optimization

This study employs the Ray Tune automated tuning tool [59], combined with Grid Search and Bayesian Optimization, to hyperparameter tune DSFA-SwinNet. Based on the initial values, the hyperparameters are fine-tuned, including training batch (batchsize), learning rate (lr, momentum), and loss function weights (flooding, α, β, γ, and loss weights of the feature maps produced by the bottleneck structure and each layer of the decoder [$W_1$, $W_2$, $W_3$, $W_4$, $W_5$]) in turn. Among them, flooding comes from the study of Ishida et al. [60], which aims to prevent training overfitting.

The search space for hyperparameters is shown in

Table 3. Hyperparameter search initial values, algorithms, and search space definitions. Grid search is denoted as GS and Bayesian optimization is denoted as BO.

Param	Search Space	Algorithm	Initial Values
batchsize	{2,4,8,16}	GS	4
flooding	[0,1]	BO	0.4
$[W_1$$, W_2$$, W_3$$, W_4$$, W_5$]	[0,1]	BO	{1,1,1,1,1}
α, β, γ	[0,1]	BO	{1,1,1}
lr	[0.0001,0.1]	BO	0.0001
momentum	[0.1,0.9]	BO	0.9

. A total of 1250 instances are randomly selected from the training set of the Google subset to create a small dataset for hyperparameter search experiments. The dataset is subsequently split into training and validation sets in a 75%/25% ratio, resulting in 1000 instances for training and 250 for validation. To ensure the adequacy of parameter exploration and optimization effect, the number of rounds of each search is set to 30 epochs based on the results of the pre-experimental runs of the experimental dataset. The convergence speed and IoU are comprehensively considered to select the optimal hyperparameters, and the initial values are updated.

Figure 9 presents the correlation between different values of hyperparameters and IoU during the training process. Additionally, variance analysis evaluates the sensitivity of IoU to each hyperparameter, with the results shown in Figure 10. Among the 12 hyperparameters of interest, $W_4$, $W_1$, lr, and $W_5$ are ranked by their significant impact on IoU, indicating that the deeply supervised training method effectively enhances the ability of the model to learn multi-scale features. Figure 9d–g further demonstrate that although flooding, α, β, and γ do not have a significant direct impact on IoU, their adjustments help accelerate the convergence speed of the model, and in some cases, there are local optimal intervals.

3.3.3. Experimental Parameter Settings

During training, the Stochastic Gradient Descent (SGD) optimizer is utilized along with Cosine Annealing and Label Smoothing techniques, and the model is trained for 100 epochs to obtain the best parameters.

Table 4. Training hyperparameters.

Param	Value
batchsize	16
lr	0.0030
flooding	0.0528
momentum	0.8000
weight_decay	0.0001
α	0.0419
β	0.6464
γ	0.6866
$W_1$	0.0000
$W_2$	0.6626
$W_3$	0.4180
$W_4$	1.0000
$W_5$	1.0000

shows the hyperparameters of the training process.

3.4. Comparison Evaluation

To further validate the effectiveness and practicality of DSFA-SwinNet for PV area extraction, this study compares it with several well-known segmentation methods, including Swin-Unet [61], TransUnet [62], ACCoNet [63], Unet [35], MP-ResNet [64], DeepLabV3+ [33], MAResU-Net [65], CMTFNet [66], LETNet [67], PIDNet-L [68] and BASNet [69]. Both DSFA-SwinNet and Swin-Unet utilize the Swin Transformer as their backbone encoder, and they are similar to TransUnet, which combines Transformer and CNN. All three architectures can effectively handle long-term dependencies. Unet and DeepLabV3+ have been employed for architectural segmentation in HRSI, and their performance and robustness have been well proven in complex environments. MP-ResNet utilizes parallel multiscale branches to capture semantic context effectively, significantly expanding its valid receptive fields and enhancing the embedding of locally discriminative features. ACCoNet and MAResU-Net enhance the segmentation performance of the model in complex remote sensing scenes by integrating an attention mechanism to capture contextual information. CMTFNet specializes in leveraging multiscale features and graph structures to improve segmentation accuracy while maintaining high computational efficiency. BASNet improves the accuracy of edge segmentation by designing a bilateral self-attention module to effectively capture global contextual information. LETNet constructs the Lightweight Dilated Bottleneck (LDB) module and the Feature Enhancement (FE) module to enhance the ability to capture local feature details. PIDNet-L is a novel three-branch network that efficiently parses detailed, contextual, and boundary information, leveraging boundary attention to guide the fusion of detailed and context branches. Through comparison with these established methods, the performance and potential of DSFA-SwinNet in extracting PV areas from HRSI will be further assessed, while offering valuable insights and guidance for both research and practical applications in this domain. To ensure a fair comparison, each model is configured with identical training hyperparameters and the same loss function as DSFA-SwinNet.

Table 5. Model parameter setting. Number of Classification Head refers to the number of prediction results from different layers or branches of the model that are involved in the loss function calculation during training.

Model	Input Size	Number of Classification Head
Swin-Unet	256 × 256	1
TransUnet	256 × 256	1
ACCoNet	256 × 256	5
Unet	256 × 256	1
MP-ResNet	256 × 256	2
DeepLabV3+	256 × 256	2
MAResU-Net	256 × 256	1
CMTFNet	256 × 256	1
BASNet	256 × 256	3
LETNet	256 × 256	1
PIDNet-L	256 × 256	3
DSFA-SwinNet	256 × 256	5

provides the specific parameter configurations for each model.

Table 6. Quantitative comparison (%) of the BDAPPV (Google) dataset.

Model	Precision	Recall	F1	IoU
Swin-Unet	72.91	89.23	78.64	67.05
TransUnet	86.73 1	94.67	89.8	82.81
ACCoNet	44.34	98.47	58.48	43.95
Unet	82.31	95.45	87.62	79.47
MP-ResNet	81.52	94.48	86.62	78.03
DeepLabV3+	86.24	95.82	90.28	83.25
MAResU-Net	84.91	95.42	89.86	82.55
CMTFNet	80.80	94.56	86.04	77.33
BASNet	86.99 2	95.26	90.32	83.41
LETNet	82.50	95.45	87.73	79.40
PIDNet-L	78.69	94.58	84.83	75.25
DSFA-SwinNet	85.97	96.56	90.44	83.50

¹ Underlined text highlights the second-best value in each column. ² Bold text marks the highest value in each column.

displays the quantitative comparison results on the Google subset. BASNet achieves the highest Precision (86.99%) with its multi-scale residual refinement module (RRM) and ranks second in F1 (90.32%) and IoU (83.41%). Compared to BASNet, DSFA-SwinNet has higher F1 and IoU values of 0.12% and 0.09%, respectively. ACCoNet, despite recording the top Recall value of 98.47% through its adjacent context coordination modules (ACCoM), lags behind the other models in Precision (44.34%), F1 (58.48%), and IoU (43.95%). Compared to other models, DSFA-SwinNet excels in PV area segmentation, ranking as the best performer in F1 (90.44%) and IoU (83.50%), and is only 1.91% lower than ACCoNet in Recall (96.56%).

Figure 11 provides an example extraction on the Google subset. In Figure 11a, under various lighting conditions, Swin-Unet, TransUnet, ACCoNet, MP-ResNet, LETNet, and CMTFNet struggle to precisely delineate the PV areas within shadows. When the color of the PV areas is similar to the background, in Figure 11b, all models except DeepLabV3+ and DSFA-SwinNet exhibit varying degrees of missed detections. In Figure 11c–f, where the PV areas are darker than the surroundings, Swin-Unet, ACCoNet, and CMTFNet frequently misclassify roofs, walls, and glass as PV areas. Moreover, when addressing the scenario in Figure 11f with narrow gaps, DSFA-SwinNet displays superior boundary segmentation capabilities compared to the comparatively high-performing TransUnet, DeepLabV3+, and MAResU-Net. Overall, DSFA-SwinNet shows better robustness and accuracy in segmenting diverse PV shapes.

Table 7. Quantitative comparison (%) of the Jiangsu PV dataset (PV03).

Model	Precision	Recall	F1	IoU
Swin-Unet	94.10	94.67	94.00	89.43
TransUnet	95.15	96.36	95.52	91.84
ACCoNet	95.74 1	95.65	95.43	91.72
Unet	95.17	96.14	95.37	91.64
MP-ResNet	94.86	96.12	95.48	91.76
DeepLabV3+	94.69	96.81	95.52	91.81
MAResU-Net	95.27	96.25	95.55	91.89
CMTFNet	94.57	96.05	94.95	91.02
BASNet	95.83 2	95.69	95.56	91.92
LETNet	94.85	95.94	95.39	91.62
PIDNet-L	94.52	95.62	94.68	90.65
DSFA-SwinNet	95.24	96.42	95.57	92.00

¹ Underlined text highlights the second-best value in each column. ² Bold text marks the highest value in each column.

displays the quantitative comparison results on the PV03 subset. On the PV03 subset with more balanced positive and negative samples, DSFA-SwinNet ranks the top metrics for F1 (95.57%) and IoU (92.00%), places second for Recall (96.42%), and is third for Precision (95.24%). In addition, compared to the quantitative results on the BDAPPV dataset, ACCoNet shows a significant improvement in performance on the PV03 subset, which contains a balanced combination of positive and negative samples. This suggests that the impact of sample proportion on the performance of ACCoNet may be more pronounced than that on the performance of DSFA-SwinNet.

Figure 12 provides an example of the extraction on the PV03 subset. In Figure 12b, all models except DeepLabV3+, MAResU-Net, and DSFA-SwinNet incorrectly classify plants and roofs with colors similar to the PV areas as PV panels. In Figure 12d, DSFA-SwinNet is the only one, apart from the others, that accurately identifies the PV gaps between the panels. Furthermore, DSFA-SwinNet also excels over other models in segmenting boundaries, as demonstrated in Figure 12a,c,e. These results collectively indicate that DSFA-SwinNet possesses higher accuracy and refinement in the task of segmenting diverse types of PV samples.

3.5. Complexity Analysis

The time complexity of DSFA-SwinNet depends on the time complexity of the Window Multi-Head Self-Attention in the backbone network. After incorporating the DWA mechanism, assuming the input tensor format is (C, H, W), where $M_i$ (i = 0, 1, …, K) represents the size of the ith window, the time complexity of self-attention for each window size in each encoder layer is $O(M_i^2{\displaystyle \frac{C^2}{K}})$. Therefore, the time complexity within each window is:

$$\sum _{i=0}^{K-1}O\left(\left({\displaystyle \frac{H}{M_i}}\times {\displaystyle \frac{W}{M_i}}\right)\times M_i^2{\displaystyle \frac{C^2}{K}}\right)=\sum _{i=0}^{K-1}O\left({\displaystyle \frac{HW}{M_i^2}}\times M_i^2{\displaystyle \frac{C^2}{K}}\right)=O\left(HW{C}^2\right)$$

(22)

The time complexity for cross-window computation is:

$$\sum _{i=0}^{K-1}O(({\displaystyle \frac{H}{M_i}}\times {\displaystyle \frac{W}{M_i}})\times M_i^4{\displaystyle \frac{C^2}{K}})=\sum _{i=0}^{K-1}O({\displaystyle \frac{HW}{M_i^2}}\times M_i^4{\displaystyle \frac{C^2}{K}})=\sum _{i=0}^{K-1}O(M_i^2{\displaystyle \frac{HW{C}^2}{K}})$$

(23)

Therefore, the overall time complexity of DSFA-SwinNet is:

$$O=4O\left(HW{C}^2\right)+2\sum _{i=0}^{K-1}O\left(M_i^2{\displaystyle \frac{HW{C}^2}{K}}\right)=4HW{C}^2+2{\displaystyle \frac{HW{C}^2}{K}}\sum _0^{K-1}{M}_i^2$$

(24)

In the complexity analysis experiments, this study assesses key performance indicators including training duration, inference efficiency, parameters, and the count of floating-point operations (FLOPs). Training duration is the average time required to complete a full training epoch on the specific dataset. Inference efficiency indicates the average time required for the model to generate predictions for the given dataset. FLOPs are determined based on the model input tensor dimensions (1,3,256,256) to gauge the computational intensity of the model. The PV03 dataset is selected as the experimental dataset and all experiments are executed on identical hardware to ensure comparable and reliable outcomes.

As depicted in

Table 8. Complexity study on the PV03 dataset.

Model	Training Duration (s)	Inference Efficiency (s)	Parameters (M)	Flops (Gps)
Swin-Unet	236.38	114.24	41.39	0.031
TransUnet	317.59	135.06	105.32	3.88
ACCoNet	2498.21	139.02	127.02	13.30
Unet	235.00	83.50	13.40	10.68
MP-ResNet	299.2	104.29	55.03	8.12
DeepLabV3+	195.92	112.28	46.62	3.99
MAResU-Net	300.00	121.90	26.28	2.61
CMTFNet	206.70	116.21	30.07	2.56
BASNet	145.71 1	83.66	12.57	1.18
LETNet	295.8	300.04	0.95	1.08
PIDNet-L	224.46	309.87	37.30	2.16
DSFA-SwinNet	168.78 2	108.38	25.88	0.99

¹ Bold text highlights the smallest value in each column. ² Underlined text denotes the second smallest value in each column.

, Swin-Unet possesses the lowest FLOPs value (0.031 Gps). LETNet has the lowest number of parameters (0.95 M). BASNet is the fastest to train, while UNet has the fastest inference. Compared to the baseline model, DSFA-SwinNet utilizes the vision Transformer property to obtain the second highest FLOPs value (0.99 Gps) after Swin-Unet without a substantial increase in the parameter count (25.88 M). On the PV03 dataset, DSFA-SwinNet places second in training time after BASNet and third in inference speed after UNet and BASNet. Compared with Swin-Unet, which employs Swin-Transformer with a fixed window as the backbone network, DSFA-SwinNet improves the training time and inference speed by 67.60 s and 5.86 s, respectively, despite incorporating additional convolutional operations.

3.6. Ablation Experiments

3.6.1. Ablation Experiments on Model Components

To assess the efficacy of DSFA-SwinNet, this section examines how various components influence its performance, including the PAR bottleneck structure, DSFA mechanism, refined skip connection approach, MLUH module, and DWA mechanism, through ablation experiments and analyses conducted on the Google subset.

Table 9. Ablation experiment results on model components for DSFA-SwinNet on the Google subset.

Test	Skip Connection	DSFA	PAR	MLUH	DWA	Precision	Recall	F1	IoU
1	-	-	-	-	-	76.23	89.04	80.95	69.58
2	√	-	-	-	-	79.41	94.02	84.87	75.39
3	√	√	-	-	-	83.02	94.06	88.19	79.07
4	√	√	√	-	-	83.37	94.71	88.67	80.43
5	√	√	√	√	-	84.54	95.37	89.63	82.69
6	√	√	√	√	√	85.97 1	96.56	90.44	83.50

¹ Bold text marks the highest value in each column.

presents the quantitative evaluation results from the ablation experiments of DSFA-SwinNet. Tests 1 through 5 utilize Swin-Transformer with the window size of 7 as the encoder. The model in Test 1 ranks the lowest in Precision (76.23%), Recall (89.04%), F1 (80.95%), and IoU (69.58%) metrics. Compared to Test 1, Test 2 achieves an improvement of 3.18%, 4.98%, 3.92%, and 5.81% in Precision, Recall, F1, and IoU, respectively, attributed to the cross-tier feature fusion of the refined skip connection strategy. In comparison with Test 2, Test 3 fuses the concatenation results of skip-connections from both spatial and frequency dimensions by incorporating the DSFA mechanism into the refined skip connection strategy, leading to model enhancements of 3.61%, 0.04%, 3.32%, and 3.68% in Precision, Recall, F1, and IoU, respectively. The PAR introduced in Test 4 enriches the internal information of the encoder output features by merging the last two layers of encoder features through a pyramidal structure, resulting in model improvements of 0.35%, 0.65%, 0.48%, and 1.36% in Precision, Recall, F1, and IoU, respectively, over Test 3. Test 4 exhibits a more gradual ascent in Precision and F1 compared to the other tests, primarily due to the inherently small size of high-level features when extracted at multiple scales, which restricts the potential for enhancing model accuracy. Considering that the output feature dimension of the final encoder layer is 8 × 8 based on the experimental dataset, the accuracy improvement for the model is relatively modest. In Test 5, DSFA-SwinNet with the MLUH module enhances PV areas extraction precision by integrating decoder feature maps through the deeply supervised approach during training. Contrasted with Test 4, Test 5 realizes improvements of 1.17%, 0.66%, 0.96%, and 2.26% in Precision, Recall, F1, and IoU, respectively, with notable advancements in Precision, F1, and IoU. Test 6 broadens the scope of single window constraints on PV feature extraction based on Test 5 through the DWA mechanism. Compared with Test 5, the Precision, Recall, F1 and IoU metrics of Test 6 are improved by 1.43%, 1.19%, 0.81% and 0.81%, respectively.

The results of the sample ablation experiment are shown in Figure 13. Since Swin-Transformer is originally designed for medical imaging tasks, the detection of PV areas fails to outline the boundary accurately, and the phenomenon of misdetection of other objects into PV areas occurs in example (c). The incorporation of the refined skip connection strategy notably ameliorates segmentation precision, yet instances of misdetection persist, such as in example (b). Subsequently, as shown in example (c), the integration of the DSFA mechanism renders the feature capturing ability of the model. However, the high-level features from the encoder are not optimally leveraged. To address this, the PAR bottleneck structure is introduced, as evident in example (e), which mitigates the missed detection problem. In example (a), Tests 1–5 inaccurately identify roof glazing as a PV areas. In example (b), Tests 1–5 mistakenly interpret the background roofs as PV areas to varying extents due to the elongated shape and color similarity with the background. The addition of the DWA mechanism bolsters the capability of the model to learn multi-scale representations of PV, leading to marked enhancements in resolving these misclassifications.

To more intuitively observe the learning process of PV features by DSFA-SwinNet, this study visualizes the features of a PV panel sample from the Google subset and a PV array sample from the PV03 subset based on Test 6. These samples represent two typical spatial distribution forms of PV scenes, with the results presented in Figure 14 and Figure 15, respectively.

Vertically, the high-resolution feature maps emphasize fine-grained texture features, while the low-resolution feature maps capture broader spatial patterns related to PV structures. Horizontally, the attention regions of spatial attention and frequency attention differ, and dynamic weighted fusion effectively integrates both regions through pixel-wise adaptive weighting.

In the PV panel sample, there is a large background area, and narrow gaps exist between the PV panels. As shown in Figure 14 (1, a), the output feature map of the PAR bottleneck structure performs fusion and multi-scale processing of the high-dimensional features from the last two layers of the encoder, preliminarily defining the probability distribution of the PV areas. By analyzing Figure 14 (1, b) and (1, c), as well as Figure 14 (2, b) and (2, c), it can be observed that spatial attention significantly enhances the expansion of PV features, while frequency attention focuses more on removing background noise and clearly defining the boundaries of the PV areas. In the high-resolution feature maps, such as Figure 14 (3, b) and (3, c), both attention mechanisms capture subtle texture variations in the PV areas and complement the PV features that are not predicted. Comparing Figure 14a,d, after processing with the DSFA mechanism, PV detail features are further enhanced, and background noise is significantly reduced. Additionally, by comparing Figure 14 (1, d) and (2, a), as well as Figure 14 (2, d) and (3, a), it is evident that in the refined skip connection strategy, after fusion with the higher-level features from the encoder, the distribution of the PV areas becomes more centralized, and the PV texture features become clearer.

In the PV array sample, Figure 15 (1, b) and (1, c), as well as Figure 15 (2, b) and (2, c), show that the attention regions of spatial attention and frequency attention focus on different aspects. It is worth noting that the annotation data for this sample contains some errors, as the serrated boundaries of the PV array are not annotated in detail. However, from Figure 15 (2, b) and (3, b), as well as Figure 15 (2, c) and (3, c), it can be observed that the striped texture of the PV array becomes clearer. In Figure 15 (3, d) and the final prediction result, DSFA-SwinNet model successfully focuses on the gaps between the PV panels and the serrated boundaries in the PV array, demonstrating its excellent ability to recognize the fine details of the PV structure.

In conclusion, the PAR bottleneck structure, DSFA mechanism, refined skip connection strategy, MLUH module, and DWA mechanism collectively refine extraction precision, segmentation accuracy, and overall robustness of DSFA-SwinNet model for the PV areas extraction task. They diminish the interference of complex background with PV areas detection and empower DSFA-SwinNet to adeptly discern the PV areas within HRSI. DSFA-SwinNet is fully capable of meeting the stringent requirements for PV areas detection in HRSI.

3.6.2. Ablation Experiments on Loss Functions

To alleviate the imbalance between positive and negative samples, optimize different model objectives, and learn diverse features at different levels, this study designs a weighted combination of hybrid loss functions. For each feature map, the computation involves three loss functions: WBCE Loss, Dice Loss, and Lovasz-Softmax Loss. N denotes the number of pixels in the image. The time complexity for calculating each loss term is $O\left(N\right)$. Thus, the total time complexity for the three loss functions is:

$$O(3\times N)=O\left(N\right)$$

(25)

Assuming there are k feature maps, the time complexity of the computation process is $O(k\times N)$. Since backpropagation requires the weighted values of all feature maps, let the time complexity of backpropagation be M, then the overall time complexity is $O(k\times N+M)$. In this study, the hybrid loss function contains 15 terms, but to make it more focused on the key objectives and avoid unnecessary redundant terms, 8 hyperparameters (α, β, γ, and loss weights of the feature maps produced by the bottleneck structure and each layer of the decoder [$W_1$, $W_2$, $W_3$, $W_4$, $W_5$]) are tuned. In the resulting hyperparameter combination, $W_1$ is 0, with $W_4$ and $W_5$ are 1, implying that during the training process, the focus is placed on $W_4$ and $W_5$.

Table 10. Ablation experiment results on loss functions for DSFA-SwinNet on the Google subset. W denotes WBCE Loss, D denotes Dice Loss, and L denotes Lovasz-Softmax Loss, while α, β, and γ represent the corresponding weights. “Number of Classification Head = 1” indicates that the output feature maps from the top decoder layer are used in the loss calculation. Training duration is the average time required to complete a full training epoch.

Test	Loss Function	Number of Classification Head	Training Duration (s)	Precision	Recall	F1	IoU
1	αW	5	177.15	74.39	96.19	82.34	72.10
2	αW + βD + γL	1	175.15 1	84.58	95.85	89.12	81.62
3	αW + βD + γL	5	185.13	85.97	96.56	90.44	83.50

¹ Bold text highlights the optimal value in each column.

presents the results of DSFA-SwinNet segmentation efficiency and performance on the Google subset for different loss functions and varying numbers of classification heads. Compared to Test 1, Test 3 exhibits an increase in training duration of 7.98 s after incorporating Dice Loss and Lovasz-Softmax Loss. However, this leads to significant improvements in Precision, Recall, F1, and IoU, which increase by 11.58%, 0.37%, 8.10%, and 11.40%, respectively. This result suggests that the weighted combination of these three losses effectively enhances ability of the model to capture PV features. Building on this, Test 3, by incorporating 5 classification heads in the loss function, results in an increase in training duration of 9.98 s compared to Test 2. The Precision, Recall, F1, and IoU improve by 1.39%, 0.71%, 1.32%, and 1.88%, respectively. This change implies that the use of deep supervision in the training process significantly enhances understanding of PV details by the model. The above results indicate that, in practical training, the increase in training time caused by the addition of different loss functions and classification heads is acceptable.

3.7. Cross-Subset Testing

The Google subset is sampled from Europe, while the PV03 subset is sampled from Jiangsu, China. Therefore, two sets of tests were designed in this section:

• Training on the Google subset and predicting the test set from the PV03 subset;
• Training on the PV03 subset and predicting the test set from the Google subset.

Cross-testing on datasets sampled from different environments aims to evaluate the ability of DSFA-SwinNet model to understand PV features. The segmentation results are shown in

Table 11. Cross-Subset Testing Results.

Test	Training Set	Validation Set	Test Set	Precision	Recall	F1	IoU
1	Google	Google	PV03	74.32	81.73	74.87	67.04
2	PV03	PV03	Google	68.91	94.40	79.67	66.21

, with sample results presented in Figure 16.

In Test 1, DSFA-SwinNet model trained on the Google subset demonstrates more precise detail capture of the PV panels. As shown in Figure 16b, although there is a gap error in the sample label, the model focuses on the gaps between the PV arrays, indicating that the model segments more based on PV panel features. In Test 2, Figure 16d–f reveal that DSFA-SwinNet model trained on the PV03 subset accurately identifies PV panels of varying scales and shapes. The above experiments demonstrate that DSFA-SwinNet model can effectively cope with environmental interference and learn to extract PV features.

4. Discussion

As highlighted in Table 1, this study employs two open-source PV datasets with significantly imbalanced positive-to-negative sample ratios for comparative experiments. The quantitative results presented in Table 6 and Table 7 show that models such as Swin-Unet, ACCoNet, MP-ResNet, and PIDNet-L perform better on the PV03 dataset, where the positive-to-negative sample ratio is more balanced. This indicates that these models still face limitations when generalizing to multi-scale PV area extraction in HRSI.

The proposed DSFA-SwinNet excels in extracting both PV panels and arrays, achieving the best performance on two datasets. The DWA mechanism, by incorporating multiple window sizes within the window intervals for patch cutting, enables DSFA-SwinNet encoder to capture multi-scale contextual information at each layer, overcoming the limitations of single window approach of the traditional Swin-Transformer. The Refined Skip Connection Strategy and PAR bottleneck structure optimize the feature propagation paths, ensuring that features at different scales are uniformly captured. The combined features are fed into the DSFA mechanism, where spatial and frequency domain feature information is decoupled, and background noise is filtered through dynamically generated weights. As illustrated in the heatmaps of hyperparameters flooding, α, β, and γ in Figure 9, these hyperparameters exhibit an optimal selection range. The sensitivity analysis results in Figure 10 indicate that $W_4$, $W_1$, lr, and $W_5$ have a significant impact on the IoU metric. Researchers are encouraged to adjust hyperparameters with notable effects on IoU based on the characteristics of different datasets, while selectively tuning those influencing convergence speed of the model as needed. This integration of the training strategy with hyperparameter optimization leads to substantial improvements in both model performance and training efficiency.

However, there are still failure cases in complex scenes within the PV areas detection task. As shown in Figure 17a, in samples with occlusions from trees and buildings, almost all models fail to detect the occluded photovoltaics. In Figure 17b, glass that is similar in color and texture to photovoltaics is misclassified as PV by all models. Although DSFA-SwinNet outperforms other models, failure may still occur in certain complex scenes. In the future, the synergistic relationship between PV textures and the surrounding environment will be further analyzed and refined, and an attempt will be made to integrate multi-source data fusion to improve the recognition of PV areas in such complex environments.

In the hyperparameter optimization experiment, Grid Search and Bayesian Optimization algorithms are used to select training batch, learning rate, and loss function weights. However, other hyperparameters, such as the size and stride of the convolution kernel, as well as the window size range selected by DWA, may also impact the final results, and their effects are not analyzed. Additionally, as shown in Table 8, the proposed DSFA-SwinNet still has potential for improvement in training duration and inference speed. Future work will focus on further refining hyperparameter tuning and considering the integration of advanced models like VMamba to enable optimal extraction of PV areas in a shorter time.

5. Conclusions

Detection of PV areas from HRSI is a crucial research focus within the domain of remote sensing image segmentation. However, the diversity of viewpoints of PV built-up areas poses a challenge for effective extraction of PV built-up areas.

In this study, a multi-scale PV areas extraction method (DSFA-SwinNet) is introduced that dynamically decouples the spatial and frequency domains. The refined skip connection strategy is designed that integrates the proposed DSFA mechanism, PAR bottleneck structure, and MLUH module to achieve fine-grained PV areas detection by performing multiscale representation learning from both spatial and frequency domain dimensions. In addition, the physical element constraints are combined with the dynamic window size adjustment mechanism to lift the restriction of fixed window size on Swin-Transformer backbone network, thereby enhancing computational efficiency and maintaining better extraction accuracy. To address the issue of extreme sample imbalance, a hybrid loss function is designed and an automated parameter tuning tool is applied to perform meticulous hyperparameter tuning by combining Grid Search and Bayesian Optimization, which provides a programmatic idea of the parameter ratio of the hybrid loss function for related research. Finally, rigorous and extensive performance evaluation experiments are conducted on the Google subset of the BDAPPV dataset and the PV03 subset of the Jiangsu PV dataset. DSFA-SwinNet performs well in PV segmentation while maintaining a low computational footprint (0.99 Gps) and a moderate number of parameters (25.88 M). Specifically, DSFA-SwinNet achieves 83.50% and 92.00% in IoU, as well as 90.44% and 95.57% in F1 Score, respectively, outperforming other models in the comparative experiments.

DSFA-SwinNet provides an efficient, cost-effective, and reliable method for dynamic PV areas detection based on HRSI providing more comprehensive data support for urban PV planning. Moving forward, the robustness of the model will be further enhanced, and lightweight detection networks for PV areas will be explored, contributing to the promotion of sustainable energy development.