Image-Based Gimbal Control in a Drone for Centering Photovoltaic Modules in a Thermal Image ^†

Abstract

Recently, there have been many types of research applying drones with a thermal camera to detect deteriorations in photovoltaic (PV) modules. A thermal camera can measure temperatures on the surface of PV modules and find the deteriorated area. However, a thermal camera generally has a lower resolution than a visible camera because of the limitations of cost. Due to different resolutions between the visible and thermal cameras, there are often invalid frames from a thermal camera. In this paper, we describe a gimbal controller with a real-time image processing algorithm to control the angle of the camera to position the region of interest (ROI) in the center of target PV modules to solve this problem. We derived the horizontal angle and vertical position of ROI in visible images using image processing algorithms such as the Hough transform. These values are converted into a PID control signal for controlling the gimbal. This process makes the thermal camera capture the effective area of target PV modules. Finally, experimental results showed that the photovoltaic module’s control area was properly located at the center of the thermal image.

1. Introduction

As the photovoltaic system is becoming popular as a future-oriented green energy, there is a strong tendency to install more PV systems widely. However, as operation time is increasing, the efficiency of the PV system is decreasing due to several reasons such as contamination of the PV module’s surface, back surface temperature, and shadows [1,2]. Thus, it is necessary to inspect the current status of the module surface regularly. Having a regular diagnosis of the module surface will ensure the best efficiency is kept by sending a real-time alarm for maintenance before the problem occurs.

Various previous studies have tested maintenance methods to keep the efficiency of the PV system. Conventional studies aimed to find a series of faulty modules by measuring the voltage and current of the PV module, monitoring the power generation status of the inverter [3,4], and detecting contamination cracks on the surface with a camera [5,6]. Other studies have manually inspected PV modules using thermal images or electroluminescence images [7,8,9,10]. Recently, studies have been conducted to find hotspots using a thermal camera [11,12]. Also, several studies used a drone to inspect a wider area and reduce inspection time. When using a drone-based thermal camera, there is an additional benefit: it can eliminate the hurdles of inspection caused by the problematic location of the site, e.g., on the roof of a high building or sea surface.

Since it is difficult to distinguish objects only with a thermal imaging camera, most studies use dual cameras composed of a thermal camera and a visible camera. The region of interest (ROI) is derived through image processing of the most visible image, and temperature information of the object is extracted by referring to temperature data of the thermal image. However, a thermal camera generally has a lower resolution than a visible camera because of the limitations of cost. Due to different resolutions between the visible and thermal cameras, there are often invalid frames from a thermal camera. Using a reasonable low-resolution camera, it is impossible to take an image from high altitude due to limited resolution. Therefore, the drone has to be placed at a closer distance from the PV module. In that case, since we cannot see the whole picture of the module, it is a prerequisite to take the picture in the center of our target object.

A gimbal system is commonly used for obtaining clean images from drone-mounted cameras, as drones in flight present a variety of vibration frequencies. The system consists of a structure to support a camera module and a stabilizer to block outside vibrations and keep the right angle [13]. A gimbal controls the target angle of a camera and provides advantages for acquiring better and proper images. This subject and the importance of using a gimbal on a drone was discussed by various authors [14,15]. Furthermore, a gimbal dampens vibrations, which is significantly beneficial for real-time image stabilization applications [16]. Apart from smoothing angular movements and dampening vibrations, a gimbal maintains a camera in a predefined position [17]. This is mostly the position in which the camera’s axis is horizontal or vertical, but all other positions according to the gimbal’s technical capabilities are possible.

In this paper, we propose an image processing and control system that can automatically adjust the gimbals for PV module inspection using autonomous drones. The proposed system consists of a dual camera, a gimbal frame, a gimbal control board (GCB), and a gimbal control system (GCS). The dual camera consists of a thermal imaging camera and a visible camera and uses a brushless DC (BLDC) motor for the gimbal frame. The GCB controls the motor through the pulse width modulation (PWM) input signal, and the main controller GCS implements image processing to control the angle of the thermal imaging camera in the visible camera. GCS calculates the center of the panel area and also the necessary value for angle adjustment from the image processing of captured images from a visible camera, and then sends the resulting data to the GCB. Consequently, we could reduce the time for pre-processing, and we could also increase the accuracy of inspection.

2. PV Module Imaging System Using Drone Equipped with Dual Camera

2.1. Considerations for Acquiring an Effective Thermal Image Using a Drone

Figure 1 illustrates the images captured from different altitudes. With a 100-m distance, we could obtain the full area of PV modules within one frame (Figure 1a,b), but due to low resolution in thermal camera, we could not obtain the explicit and proper images for thermal evaluation (Figure 1b). On the contrary, with a 10-m distance (Figure 1c,d), we could obtain the target image quality for thermal evaluation (Figure 1d). Therefore, it is necessary to place and keep the drone at a short distance, e.g., 10-m height.

If we refer to Figure 2a,c, which were captured with the visible camera, both look acceptable for further processing. However, if we compare the images from thermal cameras, due to using different resolutions between the visible and thermal cameras, Figure 2b could not obtain the proper ROI for further processing, while Figure 2d shows an aligned image for thermal analysis. Thus, it is important to place the target ROI of a thermal camera based on image processing from a visible camera.

Figure 3 shows the comparison of the acquired images with or without gimbal control during the drone’s flight. If the gimbal is not controlled, the drone’s driving changes irregularly due to the drone’s GPS error, vibration, wind, and influence of the surrounding environment. Unstable drone driving has a direct effect on the camera image. However, in an ideal case, a uniform image may be obtained through gimbal control in a straight line. Obtaining a uniform image through gimbal control can help substantially in the next analysis process.

Therefore, in this paper, we designed and implemented an image-based gimbal control system that is equipped with an independent dual camera mounted on the gimbal and performs image processing based on the acquired visible image to control the gimbal’s roll and tilt so that the ROI of the acquired image is horizontal and placed in the center.

2.2. Gimbal System

In this work, a three-axis brushless gimbal stabilizer alloy was applied to control the camera’s proper angle. The applied gimbal consists of three sets of BLDC motor for the function of pan, roll, and tilt, and a 32-bit microprocessor integrated with the GCB, a gimbal IMU (inertial measurement unit) with a three-axis accelerometer and a three-axis gyroscope, a wireless receiver for external PWM signal input, a slip ring for 360-degree continuous rotation, and an aluminum alloy frame for light weight (Figure 4). The full rotation axis symmetrical design is adapted for bearing and uses no virtual spaces in order to minimize the vibration. The gimbal functionality is controlled by a GCB, which calculates corrections for the camera position based on the IMU’s horizon deflection mounted on the camera holder.

2.3. GCS Equipped with Dual Camera

In this work, the GCS was developed as shown in Figure 5. The GCS is designed to control and implement the image processing for both thermal cameras with a resolution of 640 × 512 and a visible camera with a resolution of 1280 × 720. The output from the GCS is the control value for tilt motor and roll motor. With this controller, firstly, image processing is done from the image of the visible camera to find the target ROI for the thermal camera, and it controls the gimbal on a real-time basis. It is finally possible to obtain an excellent quality of the image from a thermal camera, which requires positioning the target area of the object in the center of the ROI.

2.4. Design of GCS

The block diagram of our drone system is shown in Figure 6. The proposed thermal imaging system captures the visible and thermal images of the PV modules and uses the visible image to derive the angle and center position of the PV module area. The angle of the derived module area is converted into the adjustment value for controlling the horizontal angle, and the position of the central area is converted into the adjustment value for controlling the vertical angle. The obtained adjustment values are inputted to the gimbal mechanism by calculating it with the signal of the wireless communication unit for the external wireless device.

The gimbal control method is divided into vertical control and horizontal control. Firstly, it is necessary to find the center position of the PV module through the gimbal’s vertical adjustment. If the PV module does not exist at the center of the thermal image, the thermal image with the upper and lower parts of the PV module will be out of the ROI. In this case, it is difficult to detect the hotspot area of the PV module accurately. Secondly, after having the right vertical respect adjustment, the gimbal needs to be aligned horizontally to adjust the gimbal tilt so that the PV modules in the image can be captured in the proper horizontal orientation. Although we explained the process in two steps, it works simultaneously. Therefore, the camera needs to be adjusted. Drone images are used to find the angle of the camera and the center of each module. With the horizontal and vertical angle control function, the gimbal is controlled in real-time so that the visible region and the thermal image are horizontal, and the ROI is captured at the center of the image.

2.5. Gimbal Control Process to Reach Desired Angle and Position

Figure 7 illustrates the process from image acquisition through the entire chain. The process involves (1) capturing the image from visible camera, (2) finding the PV module area using various image processing, such as threshold function, inRange function etc., (3) calculating horizontal angle and vertical position from the results of image processing, based on the Hough transform, etc., (4) calculating PID control value to obtain the proper PWM values, (5) transferring the calculated PWM values to GCB, and (6) GCB controlling BLDC motors as given values. This process will be repeated from phase 1 again as closed loop control.

3. Implementation of Gimbal Control through PV Image Analysis

3.1. Gimbal Control with PWM Signal

Manual and automatic signals for gimbal control are transmitted via GCS. For automatic signals, control values are determined by image processing, and for manual signals, control values are determined by external radio controls that are transmitted by a X8R receiver. The control value for the gimbal will be determined in the GCS with the combination of both manual and automatic input, with the priory of manual input signal. This result, with the format of the PWM signal, will be sent to the GCB from the GCS. According to the PWM signal, the GCB sends out control signals to each of its motors. The PWM is a cycle control method that controls the width of the pulse. Regularly, an output voltage value is maintained with high or low voltage values to produce a square wave output between 0 V and 3.3 V.

Figure 8 shows an example of the PWM. T refers to the cycle period, and t_on is the duration of high voltage. The duty-cycle is calculated using Equation (1).

$$\mathrm{DutyCycle}=\frac{t_{on}}{T}\times 100\%$$

(1)

The cycle period is 18 ms in our system, and the t_on range is between 0.7 ms and 2.3 ms. When T is 1.5 ms, the motor will be stopped, when it is 0.7 to 1.3 ms, the motor will be run as clockwise, and when it is 1.7 to 2.3 ms, the motor will be run as counterclockwise. Therefore, the duty cycle value is operated between 3 and 12. For the safety of the gimbal, the duty cycle is limited from 6.3% to 10.5%. Therefore, the PWM value is converted to a value between 64 and 108 by setting the cycle period value T (18 ms) to 1024.

3.2. Implementation of GCS

Figure 9 shows the implemented GCS in our study. The components of the CGS consist of the object with the gimbal, the embedded GCS with dual cameras, and the gimbal control/driving module. The GCS is implemented based on the hardware of Raspberry Pi 3 b+.

The thermal camera and visual camera are attached to the drone’s GCS. The image taken from the visual camera is processed and used to find physical areas of the modules. Through image processing, the GCS calculates control values through module position and angle, and converts them into PWM signals which will be transmitted to the GCB. The motor control to adjust the gimbal tilt and roll is performed by sending out independent PWM signals (PWM1 to control roll, PWM0 to control tilt).

3.3. Implementation of Object Recognition Image Processing Program

The program is implemented by OpenCV and the C++ language, which is used in the Linux Codeblocks environment. The implementation process of the image processing algorithm is as follows. We convert RGB images to Hue-Saturation-Value (HSV) images. The HSV color space is more suitable for object detection than the RGB color space. The RGB color space is a combination of colors that reduces the intuition when a specific color must be expressed. The RGB color space is not intuitive. However, in the HSV color space, intuitive native colors can be analyzed [18]. The PV module can be detected with the color of the blue hue area in the HSV color space when the PV module is detected. We extract the PV module area using OpenCV inRange function. The inRange function extracts pixels corresponding to the blue region in the HSV color space.

In extracting the PV module area, blue pixels can be extracted in some background areas as noise. To eliminate this kind of noise, dilate/erode and medianBlur function were applied. Then, we binarize the given PV module area, and using the Hough transform, we calculate the horizontal angle. The Hough transform is used in automated analysis of digital images. All PV modules contain white lines in a specific grid-like pattern. These lines are the distinctive feature of a PV module. The Hough transform is a popular feature extraction technique widely used to detect lines in images. The Hough transform detects a line represented by Equation (2).

r = xcosθ + ysinθ

(2)

where r represents the distance from the origin to closest point on the straight line, and θ represents the angle between the x axis and the line connecting the origin with that closest point.

Each line in the image is associated with a pair (r, θ). For each pixel at (x, y) in an edge image, the Hough transform finds the set of all straight lines that passes through this point. The final output of the Hough transform is the accumulator matrix. This accumulator matrix contains the quantized values of r and θ. The element with the highest value indicates the most represented straight line in the image. Therefore, the value of r and θ is determined by referring to the existing information among the three values representing the highest value.

Moreover, we calculate the vertical position with the center of gravity of the block. We show the results for an implemented image processing example in Figure 10: original visible image (a), inRange function (b), threshold function (c), Hough line function (d), bold function (e), and derived result (f).

3.4. Implementation of BLDC Control Algorithm

The control method for the gimbal system’s stabilized driving device is a proportional integral derivative (PID) controller method. The PID has a small number of parameters and a simple structure. In particular, it is intuitive to change the response characteristics in response to changes in proportional gain, integration time, and derivative time [19]. Some studies examined intelligent PIDs to automatically determine the parameters [20,21]. However, in this work, the classical PID algorithm with simple structure is applied.

From the calculated horizontal angle and the vertical position, we can obtain the PWM values seen in Equations (3)–(5).

$$e\left(t\right)=y_{target}\left(t\right)-y_{cur}\left(t-1\right)$$

(3)

$$y_{cur}\left(t\right)=K_{p}e\left(t\right)+K_d\frac{de\left(t\right)}{dt}+K_i{{\displaystyle \int }}_0^{t}e\left(t\right)dt$$

(4)

$$out{2}_{PWM}=w\times y_{cur}\left(t\right)+b$$

(5)

where t is the unit time and $y_{target}\left(t\right)$ is the target value obtained by image processing at t time, which can be a horizontal angle and a vertical position, $y_{cur}\left(t-1\right)$ and $y_{cur}\left(t\right)$ are the tilt or roll motor control value at t − 1 time and t time, respectively, e(t) is the position error value, $K_p$, $K_d$, and $K_i$ are proportional, differential, and integral coefficients, respectively, $out{2}_{PWM}$ is the PWM value for tilt motor control of the main controller, w is the slope of the PWM value and position value, and b is the minimum PWM value.

The calculated horizontal and vertical positions are encoded along with the signal received from the remote control device. The relationship between the camera focus and the PV module is always fixed regardless of the tilt. Only the part cut by the image plane changes as the tilt changes. In this development, the camera tilt is adjusted around the horizontal axis of the module seen from above the PV module.

4. Gimbal Control Experiment

In order to verify the validity of the developed system, we conducted an experimental test as shown Figure 11. Our system consists of a drone system with a gimbal, GCS with dual camera, experiment panel with photo of PV module, and remote display.

The experiment process is as follows:

• Produce the PWM signal correctly in the GCS and observe the transmission to the GCB.
• Calculate the center value and horizontal angle of the PV module by image processing.
• Analyze the gimbal control and tracking convergence according to changes in target vertical-position and target horizontal-angle.

Firstly, we confirmed whether the PWM signal was well generated by the main controller and transmitted to the GCB. As shown in Figure 12, the PWM waveform period was confirmed to be stably generated at 18 ms, and the width value was changed according to the position. Also, we confirmed that the gimbal’s BLDC motor rotated properly by generating the PWM signal.

Secondly, we extracted the central and horizontal angle values of the PV module from the image. Figure 13 shows one sample case for the image processing process. The image obtained through the image processing process, the PV module region extraction image, the binarization image, the hop transform image, and the overall result derivation image were sequentially displayed. We acquired a visible image and extracted the area of PV using the inRange function. The HSV range of the inRange function is 80 to 140 for hue, 65 to 255 for saturation, and 65 to 255 for brightness. To find the module’s center position, we binarized it with a threshold of 50 gray-value and find the boundary using the contours function. We use the Hough function with a threshold of 250 gray-value to find the PV module’s horizontal line.

Figure 14 shows various angle and center point extraction experiments with the PV module.

Table 1. Experiment result for horizontal angle cases.

Case No.	(a)	(b)	(c)	(d)	(e)	(f)	(g)
Horizontal angle [degree]	20.0	6.0	−14.0	−26.6	−43.2	19.0	−1.0
Ground truth	20.0	7.0	−15.0	−34.0	−44.0	20	−1.0
Error	0	−1	−1	−7.4	−0.8	−1	0

and

Table 2. Experiment result for vertical position cases.

Case No.	(a)	(b)	(c)	(d)	(e)	(f)	(g)
Vertical position [pixel]	124	158	234	190	358	430	272
Ground truth	96	168	240	192	350	404	273
Error	28	−10	−6	−2	8	−26	−1

show the extracted values and error values for the extracted horizontal angle and center point. Compared with the ground truth, the horizontal angle showed an error of about ±8°, and the center position value showed ±28 pixels.

Thirdly, we checked that tilt and angle control was performed correctly to position the ROI in the center and horizontally capture it. In this experiment, PID control parameters (K_p, K_d, K_i) for the PWM output were set to 0.8, 0.4, and 0.01. The experimental results are shown in Figure 15 and Figure 16. The PWM is derived using the center and horizontal values of the extracted PV module. The gimbal’s tilt and roll are adjusted to look vertically at the PV module according to the PWM signal.

In the experiment, we tested whether the tilt and roll were corrected to the target position and angle when the center position and horizontal position of the target PV module were randomly changed. As shown in Figure 15, the tilt correction result shows that the vertical angle of the gimbal is adjusted to the target position when the center position of the target PV module changes to 200, 400, 100, and 350. At this time, the tilt correction error is about 6 pixels, and the PV module seen from the drone camera is positioned at the center. As shown in Figure 16, when the horizontal angle of the target PV module was changed to 43, −75, 10, and 70 degrees, the roll rotated to approximate each horizontal angle. At this time, the roll correction error was around 3 degrees, and the PV module seen by the drone camera was aligned horizontally. Through this, the ROI of the PV module is located at the center, and it is confirmed that the proper image is taken in real-time.

5. Conclusions

Herein, we proposed a gimbal control system, which can make drones capture effective frames from a thermal camera for detecting deteriorated areas of PV modules by locating a target object in the center of an image. This system involves real-time image processing and PID signal generation. In order to calculate proper angles (pitch, roll, yaw), blob analysis and the Hough transform are mainly used for center detection and angle calculation, respectively. These position and angles are converted into a PID signal. These processes can give feedback to each other and make a target PV module stable. Finally, experimental results showed that target modules can be kept in the center area under limited conditions. In the near future, we will develop a fully automatic drone system to detect deteriorated areas using this gimbal control system after verifying its performance in real environments such as hard wind, higher trees, cloudy and rainy weather, etc.