Deterioration Diagnosis of Solar Module Using Thermal and Visible Image Processing

Abstract

Several factors cause the output degradation of the photovoltaic (PV) module. The main affecting elements are the higher PV module temperature, the shaded cell, the shortened or conducting bypass diodes, and the soiled and degraded PV array. In this paper, we introduce an image processing technique that automatically identifies the module generating the hot spots in the solar module. In order to extract feature points, we used the maximally stable extremal regions (MSER) method, which derives the area of interest by using the inrange function, using the blue color of the PV module. We propose an effective matching method for feature points and a homography translation technique. The temperature data derivation method and the normal/ abnormal decision method are described in order to enhance the performance. The effectiveness of the proposed system was evaluated through experiments. Finally, a thermal image analysis of approximately 240 modules was confirmed to be 97% consistent with the visual evaluation in the experimental results.

1. Introduction

Photovoltaic (PV) modules are made up of several solar cells. When an abnormality occurs in such a cell, the cell operates like an electrical resistor. As a result, the short circuit current decreases because of the power consumption and the series circuit characteristic. This phenomenon leads to an increase in temperature and accelerates the damage of the PV cell. Eventually, this abnormal process limits the short circuit current of the faulty PV module, reducing the power generation efficiency, performance and life of the PV module [1]. When the current capacity is reduced due to shadows produced by leaves, clouds, and dust or an abnormality occurs such that one cell is completely obscured or the cell is aged, a reverse bias is applied to the PV cell. This causes a so-called hot spot [2]. Some studies have found the modules that may cause failures by the thermal analysis of PV modules using an infrared thermal camera to prevent this loss [3,4,5,6,7,8].

Suárez-Domínguez’s PV module thermal image analysis study [5] showed that the mean temperature of the PV module was 21 degrees. However, the temperatures at the two hot spots were found to be 26 degrees and 28 degrees. E. Kaplani’s study [6], confirmed that the temperature of the hot spot area exceeded 100 °C in summer. Several other studies have also suggested that thermal imaging cameras are reliable, economical, and easy methods of inspecting hot spots, and suggest that periodic inspection can lead to optimal plant operation [6,7]. However, reflection and emissivity must be considered when inspecting a PV module using a thermal imaging camera. When PV modules are inspected from the front, a thermal imaging camera sees the heat distribution on the glass surface but only indirectly sees the heat distribution in the underlying cells. Glass reflections are specular, which means that the surrounding objects, with different temperatures, can be seen clearly in the thermal image. In the worst case, this results in misinterpretations (false “hot spots”) and measurement errors. In order to avoid the reflection of the thermal imaging camera and the operator in the glass, it should not be positioned perpendicularly to the module being inspected. However, emissivity is at its highest when the camera is perpendicular and decreases as the angle increases [8].

Some solar power plants are considering installing fixed dual cameras. However, high-resolution thermal cameras are still very expensive. In order to install a large number of these expensive cameras, a large initial investment cost is required. In addition, it is necessary to establish and maintain an information processing system for multiple cameras. In recent years, in order to inspect the PV modules installed in a wide area or in a difficult accessing place, a thermal image of the PV module using a drone is taken and a thermal temperature analysis is performed [9,10]. However, these studies rely on visually inspecting the temperature distribution and finding faulty modules while viewing infrared thermal images on the monitor. Thus, a deterioration diagnosis program of a photovoltaic module is required to manage a module installed in various solar power plants in a wide area.

Thermal cameras generally measure the wavelength emitted by an object to extract temperature information. However, it is difficult to distinguish objects according to temperature conditions. The thermal image alone provides the temperature value of a specific object. Nevertheless, object boundaries cannot be distinguished by the thermal image alone in order to automatically extract the region and determine the abnormality through the temperature in the region [11]. Therefore, research has been carried out to realize object classification and object temperature distribution through a visible camera and a thermal camera. These studies express the temperature image of a specific region by performing a matching process between a visible image and a thermal image [12,13,14]. The image registration process is particularly essential in order to obtain necessary information by using multiple thermal and visible cameras. The matching process performs a geometric alignment process on the image so that data can be compared or integrated from different images. In these studies, the feature points must be extracted well for matching, and the matching relation of feature points is important. Davis [12] proposed a method of extracting background-subtraction and contour and synthesizing thermal image and visible images to extract the feature points. Conaire [11] evaluated the use of feature points by applying various methods such as four feature points, color and edge histograms, and weighted models.

Another study suggests a method of finding a plane homography that aligns the thermal spectrum with the visible spectrum. Using the obtained homography, the thermal image is image-warped to match the image [14]. In order to obtain such a precise homography, a number of correspondence points are defined in the image manually and perform a homography search process is performed that minimizes the least squared error [15]. However, the lack of correlation between the brightness intensity of the degradation image and the intensity of the visible image makes it difficult to generate an automatic correspondence point due to the brightness. Thus, a passive correspondence point designation or a specific landmark search method is required [16].

In this paper, we introduce an image processing method that can automatically identify the module generating hot spots in the solar module. In order to overcome the difficulties of generating the automatic correspondence points proposed in the previous studies, the proposed PV module analysis system adopts the image processing method that uses the characteristic elements of the PV module. Additionally, the proposed method extracts temperature information automatically and determines whether there is an abnormality through the rectified module image. The proposed method consists of four major image processing methods using the unique features of the PV module. The first method uses the shape and color characteristics of the PV module. The second exploits the fact that the PV modules are installed in multiple clusters. The third method involves the extraction of characteristic points from the visible image and deterioration. Finally, the fourth method derives a homography transformation equation for matching between minutiae points. To extract the feature points, we use the maximally stable extremal regions (MSER) algorithm, which computes the depth change value with the neighboring pixels and generates a tree up to the neighboring pixel level value [17]. Next, the proposed method derives the area of interest by using the inrange function using the blue color of the PV module. We propose an effective matching method for feature points and a homography conversion technique using the random sample consensus (RANSAC) algorithm [18]. The module area is rectified to increase the ease of the temperature distribution inspection of the module area after the matching of the thermal image and the visible image is completed.

The temperature information is extracted from the rectified thermal image and the failure prediction module is determined by calculating the abnormal area of high temperature area. Through this image processing method, it is possible to automatically identify the module generating the hot spot of the photovoltaic module, and it can help identify the cause of hot spots by displaying the thermal image and visible image generated by the hot spot. If there is contamination, shadows, or breakage, etc., in the visible image in the high temperature area, it is easy to identify the cause. However, if there is no abnormal point, it can be judged that the performance degradation is due to the characteristic change. In order to distinguish between the normal and abnormal modules, accurate region extraction of the photovoltaic module must proceed. If the photovoltaic module area is well extracted from the thermal image, the abnormal temperature distribution is not found as shown in Figure 1a, but it is judged as normal. If the module area is incorrectly extracted as shown in Figure 1b, an abnormal temperature distribution is found and it is judged to be an abnormal module.

In Section 2, we describe an algorithm for detecting the occurrence of hot spots by analyzing the temperature distribution of a solar module. Furthermore, we describe the proposed visible image and thermal image registration, rectification process, temperature data derivation method, and normal/abnormal decision method. In Section 3, image analysis is performed on various solar modules and the effectiveness of the proposed algorithm is examined. In Section 4, we discuss the results and draw conclusions.

2. Deterioration Diagnosis Method of PV Module Using Thermal Image and Visible Image Processing

2.1. Proposed Method

The flow chart of the proposed method is shown in Figure 2.

The proposed algorithm can be summarized in the following five steps:

• Image acquisition.
• Solar module area search and block segmentation.
• Extraction of visible image and thermal image feature points and corresponding point matching.
• Matching and rectification based on corresponding points.
• Determining the abnormal module.

Figure 3 and Figure 4 show the algorithm proposed in this paper which includes the process of feature points extraction, feature points matching, segmentation, rectification and abnormal module judgment. The detailed description of the proposed method consists of following functional steps.

• A visible image (Img1) and a thermal image (Img2) are acquired by simultaneously photographing the PV module with a visible camera and a thermal camera.
• The region of interest corresponding to the region of Img1 is RImg1.
• Extract the PV module area from Rimg1 through the inrange function. The binary image represented by white in the module area portion is RMImg1.
• The feature points (MP1: mP11 to mP1n) corresponding to the vertexes of the box area are derived by the MSER algorithm in Rimg1.
• The feature points (MP2: mP21 to mP2n) corresponding to the vertexes of the box area are derived by the MSER algorithm in Img2.
• The effective matching point set (M1,M2) is derived from MP1 and MP2 through a decision criterion function.
• The homogrphy (H1) to convert M2 to M1 is found.
• Img2 is projected on Rimg1 to obtain a matching image (Rrimg).
• The temperature distribution calculation and module recognition process for each module consists of the following steps.
• In RMImg1, MB1,…, MBn are obtained by segmenting by block using the findcontuors function.
• The homography Hb1 for rectifying the region of MB1, and the homography Hbn for rectifying the region of MBn are obtained. This homography is applied to obtain each rectified image RB1,…, RBn.
• The rectified image RBImg is used to determine whether there is an abnormality through the temperature distribution and the abnormality determination equation for each module area.

2.2. Image Acquisition

To acquire the image, we constructed a stereo camera with a visible image camera (resolution: 1280 × 720) and an infrared camera (640 × 512) as shown in Figure 5. Camera calibration was performed to derive a common image area with different angles of view between the visible image and the thermal image. Through the camera calibration process, the lens distortion correction and the main shooting distance were taken at positions of 8 to 18 m, and a strategic area of the thermal image was set in the visible image area.

2.3. PV Module Area Search and Block Segmentation

2.3.1. Extraction of Region of Interest through Color Inrange

We implemented an algorithm that derives the area of the photovoltaic module by exploiting the fact that the main color is in the blue region due to the characteristics of the PV module. The color image was transformed into the HSV image using the color inrange function. The hue value was 80~140, the saturation value was 65~255, and the value was 65~255. An example of the image processing using HSV conversion and inrange function is shown in Figure 6.

2.3.2. Block Segmentation through Find Contours

Since PV modules have a cluster array structure for serial-parallel connection and the installation type may be changed in units of cluster blocks, the image was segmented in units of blocks. An example of the block segmentation by the findcontours function is shown in Figure 7.

2.4. Module Feature Point Detection Using MSER Algorithm

This step converted the visible image into a gray image and extracted the module area using the MSER algorithm. Additionally, the thermal area image was extracted using the MSER algorithm. As an example, the results of using MSER without considering module size are shown in Figure 8.

The MSER rectangular areas of the visible image as shown in Figure 8a were sorted in ascending order, and are used as the predicted area value of one PV module based on the intermediate area value. In Figure 9, we show a graph of rectangular area vs. rank by area. MSER rectangular areas within 20%, the appropriate value according to various experiments, are derived by comparing these with the predicted area values. The calculation was performed in following steps.

• Arrange rectangular area elements by area.
• Find the median predicted area of the sorted result.
• Among the median predicted areas, the rectangular regions are filtered with those of a prediction area of ±20%.

In Figure 10, we show predicted areas by using the median value. The number of green rectangles in Figure 10 is reduced, compared to Figure 8, as a result of applying the MSER algorithm considering the module size and extracting the feature points. The yellow dots are used as feature points.

2.5. Valid Matching Point, Homography and Registration

2.5.1. Valid Matching Point

An unnecessary matching point of a rectangular area is estimated as shown in Figure 10a,b. In order to remove the unnecessary matching points, the validity between the points of two images is determined, which is expressed as:

$$x{1}_i\in MP1,x{2}_j\in MP2,i:1~MP{1}_{size},j:1~MP{2}_{size}$$

(1)

$$\begin{array}{c}r=\sqrt{{\left(x{1}_i-x{2}_j\right)}^2+{\left(y{1}_i-y{2}_j\right)}^2}\\ if\left(r<t{h}_r\right)thenM{1}_k=MP{1}_i,M{2}_k=MP{2}_j\end{array}$$

(2)

The set of feature points of the visible image and degradation images are called MP1 and MP2, respectively as shown in Figure 3. The expressions $\left(x{1}_i,y{1}_i\right)$ and $\left(x{2}_j,y{2}_j\right)$ respectively represent the x value and the y value of the i-th feature point of the MP1 and the j-th element of the MP2, Equation (1) performs an iterative search while changing to the introduction number. The validity evaluation step computes the distance value (r) between the two points. If the estimated distance is smaller than the boundary value (th_r), these two points are determined as an element of the effective feature point set (M1, M2).

2.5.2. Homography Derivation through Valid Feature Points, Registration

A set MP1 of points ($x{1}_i$) existing in the projection plane space corresponding to the set MP2 of points ($x{2}_i$) existing in the two-dimensional projection plane space can be projected, and a projective transformation between the two images can be expressed as a relation function. Projectivity is defined as an invertible mapping h, which is a linear transformation through a non-singular 3 × 3 matrix [19].

$$\left(\begin{array}{c}{x}_1^{\text{’}}\\ x_2^{\text{’}}\\ x_3^{\text{’}}\end{array}\right)=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)$$

(3)

$$X^{\prime }=HX$$

(4)

The homography converting the thermal image feature point M2 into the visible image feature point M1 was obtained as follows.

$$H1:M2\to M1$$

(5)

Homography (H1) in the case of Figure 7 is as follows.

$$H1=\left(\begin{array}{ccc}1.011490762& -0.0011793& -0.17881936\\ -0.0074779& 0.95735177& 5.3210135\\ -0.0000029& -0.0000903& 1\end{array}\right)$$

(6)

Using the obtained homography H1, the thermal image is projected onto the visible image. Figure 11 shows a composite image of a red image with a visible image converted to green. The disparity between the images is shown by a simple synthesis before the conversion in Figure 11a,c. However, it can be confirmed that the images are well matched without any difference value in the composite image using the homography.

2.6. Rectification, Thermal Data Extraction, and Determination of Results

The module region was rectified to increase the ease of the temperature distribution inspection of the module area, once the registration of the thermal image and the visible image was completed. The faulty module was determined by calculating the high temperature area. Through this image processing procedure, the module generating hot spot in the PV module was automatically identified. This helped to identify the cause of the hot spot by displaying the thermal image and visible image generated by the hot spot. Visible images can be used to distinguish between contaminants, shadows, breakages, microcracks, or cell deterioration.

2.6.1. Homography Derivation and Rectification

In the blocks obtained through the segmentation, the entire area of the PV module was obtained, and the homography was derived using the edges of the valid feature points needing to be rectified.

$$HBn:MBn\to RBn,\hspace{1em}n:1~{MB}_{size}$$

(7)

$$RBimg=\sum _{n=0}^{blocksize}RBn$$

(8)

HBn uses a homography matrix for rectifying the sub-block MBn into the rectified block RBn, and obtains the rectified image Rbimg by summing the rectified images of sub-blocks. An example of obtaining the rectified images from sub-blocks is shown in Figure 12.

2.6.2. Extracting Temperature Information and Determining the Abnormality

In order to determine whether an error had occurred in the PV module, the average temperature value (T_avg), the maximum temperature value (T_max), and the minimum temperature value, (T_min) information in the module area were extracted. Next, the high threshold temperature (Th_high) and the low threshold temperature (Th_low) were determined in the PV module area. If the temperature value was less than Th_high and less than Th_low, the count value (a_count) would be increased. If the value a_count exceeded 0.2% of the module area value (area_module), then the module would be considered as an abnormal module. Figure 13 shows an example of extracting temperature information and determining the abnormality. Figure 13a shows the temperature distribution for the abnormal module among several modules, and Figure 13b shows the results of the normal module (left) and the abnormal module (right) obtained through the proposed equation.

TH_high = T_avg + T_max × 0.2
TH_low = T_avg + T_min × 0.2
if ((T_val > Th_high) or (T_val < Th_low)) a_count ++;
if (a_count > area_module × 0.002) abnomal PV module

(9)

3. Experiment

To evaluate the validity of the proposed algorithm, 40 visible and thermal images were simultaneously captured using a drone. The shooting altitude was set to about 10 m, and the acquired image is shown in Figure 14.

In order to evaluate correspondence consistency, ground-true images were manually generated for seven images as shown in Figure 15.

For each image, feature points were derived, and the homography for matching was derived, projected, and finally, matched. To obtain the validity, the homography was applied to the ground-true image to calculate the match rate. The results showed an average of 97% agreement.

The mean area value of the module area obtained by the MSER algorithm was 10,336 pixels, and the standard deviation of the PV area was 215 pixels. The average area accuracy of the module area was about 98%. Finally, the proposed method determined whether the module was normal or faulty, and estimated the faulty module by calculating the high temperature area. The thermal image analysis of approximately 240 modules was confirmed to be 97% consistent with the visual evaluation. The module with the error was confirmed to be faulty due to the reflection of sunlight.

Figure 16 shows the experimental results for finding the abnormal module. Based on the GPS information, the analysis results are shown on Google Maps in Figure 16a. A green mark indicates a normal module and a red mark indicates an abnormal module. When an user presses a red mark on a screen, the user can see the detailed analysis such as the corresponding visible image (Figure 16b), thermal image (Figure 16c), merge image before registration (Figure 16d), merge image after registration (Figure 16e), normalized image (Figure 16f), and the abnormal region derivation result by the determination algorithm (Figure 16g) are expressed. As a result of pressing the mark manually, it was confirmed that both normal and abnormal modules can be distinguished well.

4. Conclusions

In this paper, we introduced an image processing technique that can automatically identify the modules generating the hot spots in solar modules. The proposed PV module analysis system adopted the image processing method that uses the characteristic elements of the PV module in order to overcome the difficulties of generating the automatic correspondence point presented in the previous studies. Additionally, the proposed method determined an abnormality of the PV module. The image processing method using the characteristic elements of the PV module extracts the feature point derivation from the visible image and the degradation image by using the shape characteristic of the color module, the rectangular shape of the PV module, and the installation of a plurality of clusters, and a homography transform equation is derived from the matching points. To extract the feature points, we used the MSER algorithm, the inrange function, an effective proposed matching method and a homography conversion technique using the RANSAC algorithm.

Experimental results show that the match between the thermal image and the visible image was 97%. The accuracy of applying the MSER algorithm and the average area derivation algorithm was confirmed to be 98%. Similarly, through the rectification of the module area, the ease of the inspection of the temperature distribution of the module area was enhanced, and the failure prediction module was determined by calculating the abnormal area of high temperature. The thermal image analysis of approximately 240 modules was confirmed to be 97% consistent with the visual and visual evaluation. Through the application of these algorithms, it is possible to automatically identify the module generating the hot spots in the solar module, and to identify the cause of the hot spots by displaying the thermal image and visible image generated by the hot spot.