A Computer Vision Line-Tracking Algorithm for Automatic UAV Photovoltaic Plants Monitoring Applications

Abstract

In this paper, the authors propose an UAV-based automatic inspection method for photovoltaic plants analyzing and testing a vision-based guidance method developed to this purpose. The maintenance of PV plants represents a key aspect for the profitability in energy production and autonomous inspection of such systems is a promising technology especially for large utility-scale plants where manned techniques have significant limitations in terms of time, cost and performance. In this light, an ad hoc flight control solution is investigated to exploit available UAV sensor data to enhance flight monitoring capability and correct GNSS position errors with respect to final target needs. The proposed algorithm has been tested in a simulated environment with a software-in-the loop (SITL) approach to show its effectiveness and final comparison with state of the art solutions.

1. Introduction

In the last few years, the PV sector has witnessed a very rapid increase in its production rate with the complicity of economic incentives offered by many countries and a general market attention toward renewable energy sources.

Maintenance represents nowadays a key aspect for PV plants profitability: in fact, many studies have shown that a very large number of PV modules present damages. In [1], it has been shown that in some critical plants the impact of failures can even reach the 80% of the total installed modules. Among all the possible failures of PV modules, the superficial ones are the most common [2]. Some studies have been calculated that the impact of defects on PV modules can increase the levelized cost of energy of around 5% [3].

One of the first tests that has been introduced for detecting the defects is the visual inspection: it can detect several defects, from delamination, to cell corrosion, from glass breakage, to the snail trails phenomenon [4]. The visual inspection performed by an expert is very effective in detecting defects. During the inspection it is possible to perform other analysis on the modules, for example, the detection of backsheet damages, but it generally requires a lot of time. Due to this reason, this technique is suitable for small–medium size ground plants, whereas for large scale ground plants, or for rooftop plants, it is almost unsuitable.

A very accurate defect identification technique is the electroluminescence test: it is performed in the laboratory, but it is able to easily detect the defects [5].

Another possible technique for failure identification on PV plants is the analysis of the power output data. This method is very simple because it can be performed with the standard remote sensing data that are almost always acquired by inverters. On the other hand, this method has two main disadvantages: for detecting the defectiveness rate it is required to have a reference power production, and it is not able to detect the position of the defects inside the plant [6].

For improving the inspection for large-scale plants or even more rooftop plants, Unmanned Aerial Vehicles (UAV) technology has been proposed. The main advantage of this technique is the very low time required in the acquisition, but it creates a large amount of raw data that should be processed [7].

Adopting this technology, it is possible to easily acquire two main different kinds of information that can be exploited together for performing a proper inspection: The first type of information is the picture of the module in the visible spectrum of the light, and the second one is the infrared imagery. This last information is widely adopted to diagnose faults in PV modules because it can detect in a easy way a wider range of defects [8].

The images acquired from the UAV can be processed both by an expert user or by computational techniques for the image postprocessing [9]. This last type of analysis is very promising because it is able to process a large amount of data in a fast and effective way to keep the entire process competitive. The analysis of the big data acquired by the UAV can be carried out in several ways: The most advanced techniques regards the analysis of the healthy status of each module: this analysis has been performed exploiting deterministic algorithms [10] or artificial intelligence techniques such as deep learning [11].

In the automated image processing techniques there are two very important aspects that should be taken into account. First, for large PV plants the access to the final information can be very hard and, thus, can slow down the overall maintenance operations. For overcoming this problem, advanced processing methods have been introduced to automatically index the information in a database of the PV plant and for providing a better user interface [12].

Second, the UAV flight must be clearly compliant with the requirements in terms of accuracy and resolution of the post processing phase. In particular, the images should be properly oriented and acquired in a predefined altitude range. The flight control system of the UAV should allow the acquisition of accurate pictures in visual and thermal spectrum, in a repeatable and effective way.

The required resolution for the postprocessing is a key factor for designing a correct flight plan: to achieve the resolution required for detecting defects, a proper trade-off between flight altitude and UAV sensor accuracy is needed [13,14]. Typically, a low flight height is preferred, giving problems in management of the UAV path planning.

The objective of this paper is to develop a real-time correction technique for the flight path of a multirotor UAV for PV plants inspections using information coming from additional sensors. In particular, cameras are particularly suitable for this kind of operations. Thus, a vision-based guidance law has been developed and tested in an appropriate simulation environment. Preliminary results regarding a possible tuning and the performance of the algorithm over a single straight line have been presented in [15]. In particular, the vision algorithm has shown capable of reconstructing the distance between the line and the UAV with a mean error of 5 cm.

In the literature, the problem of controlling a vehicle, or, in general, a robot using feedback information coming from a vision sensor, is referred to as visual servoing [16]. In particular, referring to on-board visual systems there are two ways to implement vision-feedback robot control: the dynamic look-and-move system and the direct visual servo system.

The dynamic look-and-move system performs vision-feedback control of robot manipulation in a hierarchical manner. The higher-level vision system provides a desired pose reference input to a lower-level robot controller which then uses feedback from the joints to internally stabilize the robot [17].

In direct visual servo control systems, instead, a visual controller replaces the robot controller and directly computes inputs to the robot joints. In UAVs, a direct visual servoing problem concerning the tracking of power lines has been faced in [18]. A similar solution for the tracking of linear features has been also proposed in [19]. In [20,21], similar works have been conducted on fixed-wing UAV for alignment with ground runway lines for landing.

Note that direct visual servoing approach requires an ad hoc design of the control law and, consequently, it is not compatible with pre-existing flight control software. In contrast, this work aims to develop a general law which could be implemented on different platforms. In this context, indirect visual servoing (dynamic look-and-move system) could represent a suitable solution. The idea would be the one of using the high-level vision information to provide a desired modification in the position reference to the flight control software available on the platform.

To the best of the author’s knowledge, this kind of approach (and UAV visual servoing more in general) has never been used on photovoltaic plants.

The work presented here is a very important step toward autonomous flight and, thus, opens up a scenario of complete autonomous monitoring techniques for energy plants.

The paper is structured as follows. In Section 2, the motivations behind this work are discussed, whereas in Section 3 the proposed solution is presented and analyzed. In Section 4, the case study and the simulation environment are presented and in Section 5 the results are shown and analyzed. Finally, in Section 6, some conclusions are drawn.

2. Motivations

UAV flight control for plant inspection can be carried out in two different ways: manually or in a fully automatic way. In the former, the commands to the UAV are provided by an operator through a radio controller, whereas in the latter, they are inserted in the form of waypoints on a georeferenced map (Figure 1) and tracked by the UAV exploiting the Global Navigation Satellite Systems (GNSS).

However, in this case, no guarantee that the drone will actually fly over the PV panels exists. First of all, the waypoints can be misplaced by the operator and/or not exactly corresponding to the desired location due to possible incorrect georeferencing of the map in which the waypoint itself is positioned. Note that georeferenced maps obtained from satellite imagery, e.g., Google Earth, widely used in this framework, are not a complete reliable source of information. Researches in the literature have shown that the error in planimetric coordinates (North, East), between objects as captured in Google Earth and obtained from a base map produced by ground surveying techniques, ranges from 5 [22] to 10 m [23] and even more [24,25], depending, among many factors, on the geographic location. The accuracy of measured distances in Google Earth is considerably better than the accuracy of measured individual coordinates, but, in any case, an error of 2 or more meters can be expected [26].

Second, the intrinsic inaccuracy of the received GNSS signal can lead to poor performance in the estimation of the UAV position, leading to tracking errors and consequent degradations in the inspection performance. The GNSS error sources could be many, ranging from ionospheric and tropospheric delays, clock-related and satellites’ orbit-related errors, multipath errors, and the receiver noise itself [27]. In particular, the GNSS receivers embedded on a professional UAV have been tested in [28], highlighting a 1 to 2 m and 4 m accuracy, respectively, on the horizontal and vertical position. Note that this result provides some indications about the magnitude of the error, but it cannot be generalized as it depends on the hardware used, satellite coverage, and geographic position.

The aim of an inspection procedure is gathering data from which it is possible to obtain information on the health status of each module. This information is usually captured through infrared (IR) and visible images. From the images, the most important defects which affect the performance of the module and of the power plant are recognized. In the context of UAVs for aerial inspection, the ground resolution of the images is of paramount importance. In [29], an overview of the required ground resolution for many of the most common defects is provided. From the type of defect, the required ground resolution is established (the number of pixels for each $c{m}^2$ of ground). More precisely, the resolution is influenced both by the cruise altitude of the UAV by the resolution of the camera as highlighted in Equation (1),

$$R_{ground}=\frac{R_{photo}}{H}·\frac{f}{d_{picture}},$$

(1)

where $R_{ground}$ is the ground resolution and its function of the cruise altitude H, and for the camera data: the picture resolution $R_{photo}$, the focal distance f, and the picture distance $d_{picture}$.

For identification of some defects, especially in the visible spectrum, a high sample ground resolution is usually required.

Table 1. Example of required ground resolutions for some typical defects.

Defects	${\mathit{R}}_{\mathit{ground}}$ [pixel/cm]
Snail Trail	8–10
Dirty	3–5
White Spot	2–3
Encapsulant discoloration	2–3
Other	6–8

shows the ground resolution required by some defects after a large trial campaign: it is possible to see that snail trail is the most critical defect that can be identified.

This means a lower cruise altitude and, thus, a small number of PV modules contained in each picture. Consequently, a small positioning error of the UAV can lead to captured images which contain only part of the PV row to be inspected. Thus, an accurate flight control performance is required to have a satisfactory inspection outcome, therefore there exists a need of the proposed algorithm for the enhancement of automatic flight.

Having a correct and repeatable acquisition of images is particularly useful when automatic systems for image analysis are employed (as can be the case when a large amount of information is captured, as in the case of UAV inspection and monitoring applications).

3. Proposed Solution

A conventional position control loop for a multirotor UAV can be described using the block diagram in Figure 2, where x is the position, $x^0$ the corresponding set-point and transfer functions $G_x\left(s\right)$ and $R_x\left(s\right)$ represent, respectively, the position dynamics and the position controller.

If, as is common practice, position control is implemented using GNSS, then the feedback system will be affected by the measurement noise $n_{GNSS}$. The set-point $x^0$ can be either generated directly by a pilot or may correspond to a sequence of waypoints defined on a Ground Control Station (GCS) at the beginning of the mission. In the latter case, however, an additional error term has to be taken into account, namely, the error in the positioning of the waypoints, which for example may be due to limitations of the GCS interface or inaccurate georeferencing of the plant map to be inspected. In such a case, a more realistic representation of the control system, see Figure 3, should be taken into account where the actual set-point received by the control system (denoted as ${\tilde{x}}^0$) will be the intended set-point $x^0$ (the coordinate of the PV row to inspect) plus an error term $n_{MAP}$.

This block diagram is interesting as it shows clearly that the effect of GNSS measurement error and the effect of inaccurate set-point computation enter the control system at the same point, so that letting

$$F\left(s\right)=\frac{R_x\left(s\right)G_x\left(s\right)}{1+R_x\left(s\right)G_x\left(s\right)},$$

(2)

the complementary sensitivity of the position control system, the block diagram can be equivalently represented as in Figure 4, where $n_{x^0}$ represents the simultaneous effect of all the error terms.

The proposed solution is based on the assumption that additional sensors can return a (moderately) noisy measurement of $x-x^0$. In such a case, the problem under study will become the one of exploiting this information to compensate the effect of $n_{x^0}$ on the performance of the control system. In particular, a compensator with transfer function $\mathsf{\Gamma }\left(s\right)$ is designed and properly tuned to take as input the measurement of $x-x^0$, characterized by a noise term $n_{CAM}$, and to give as output the needed quantity to cancel out the term $n_{x^0}$, as represented in Figure 5.

Note that directly an ad hoc designed UAV position controller could be used to accomplish this aim. The reason behind the choice of having a dedicated compensator operating on top of the UAV’s position control loop with respect to the flight controller (represented in the block diagram with the transfer $F\left(s\right)$ function) resides in the fact that the same concept can be applied to different platforms and control units.

Among the different sensors, cameras are particularly promising as they are already available on a typical UAV involved in this kind of inspections without additional equipment costs. Consequently, this will not lead to an overall increase in the payload of the vehicle. Through the use of appropriate techniques, images allow the extraction of useful information about the relative position and orientation between the PV row and the UAV itself.

In the computer vision literature, several possibilities to extract and analyze features exist. In particular, feature descriptors algorithms like Scale Invariant Feature Transform (SIFT) [30], Speeded Up Robust Features (SURF) [31], Histogram of Oriented Gradients (HOG) [32], and Local Binary Patterns (LBP) have been used in a wide range of applications, including face recognition [33], speech resampling [34], and cancer detection [35]. However, for the application object of this paper, since the shape to be searched in the image is known a priori and very simple, i.e., a line, the extraction process can be divided into an edge detection and a simple line detection algorithm.

3.1. Edge Detection

Edge detection highlights points in a digital image where brightness changes sharply and finds discontinuities. Various edge detection algorithms exist, e.g., Roberts cross operator [36], Prewitt operator [37], Sobel operator [38], and Canny edge-detector [39]. The latter has been chosen for its simple and precise mathematical formulation and its satisfactory performance in terms of noise rejection. Its application to an aerial image of a single PV row (see Figure 6a) is shown in Figure 6b.

3.2. Line Detection

Concerning line detection, the Hough transform [40] is applied to the obtained binary template. It allows detection of the lines appearing in the image, giving as additional information their analytical form in terms of distance of the line from the upper left corner of the image and angle between the line and the vertical.

A graphical example of its output is shown in Figure 7a.

As the aim of the algorithm is to support the monitoring process maintaining the UAV along the principal (longitudinal) direction of the PV row, only the lines which correspond to the direction of the row itself are of interest and must be taken into account in the guidance algorithm. Assuming that the relative angle between the heading of the UAV and the principal direction of the PV row is, in absolute value, lower than ${45}^{\circ }$, ambiguity in the selection of the correct lines cannot be found. This can be regarded as a reasonable assumption considering the scale of the errors addressed in this paper. Thus, the lines in the direction of the row are selected and retained, and the others are discarded. The final output is shown in Figure 7b.

Furthermore, the PV row is approximated as the mean line of the remaining lines and compared with the desired outcome. The latter is the one in which the PV row is located exactly at the center of the image with its mean line aligned with the vertical direction. The distance between the two will give an indication about the relative position in pixels between the UAV and the PV panel, indicated as $x-x^0$ in the block diagram and in Figure 8.

Finally, assuming that the width of the PV row is known, or alternatively using the altitude of the UAV for the reconstruction of the camera Field of View (FOV), the position can be converted directly into meters.

The compensator $\mathsf{\Gamma }\left(s\right)$ has been chosen so that the feedback interconnection between the complementary sensitivity of the UAV’s position control system $F\left(s\right)$ and $\mathsf{\Gamma }\left(s\right)$ is asymptotically stable and considering the case in which $n_{x^0}$ can be treated to first approximation as a constant input. Then, clearly the response to the error tends asymptotically to zero if and only if a compensator with integral action is adopted. Its transfer function can be written as

$$\mathsf{\Gamma }\left(s\right)=K_p+\frac{K_i}{s},$$

(3)

where $K_p$ and $K_i$ are respectively the proportional and integral gain. These two gains can be tuned using classical methods from linear control systems theory.

4. Test Case

4.1. Simulation Environment Architecture

For the purpose of testing the developed algorithm, a simulation Software-in-the-Loop (SITL) approach has been adopted. In particular, the developed SITL tool is based on the PX4 autopilot’s firmware [41].

In detail, the PX4 SITL communicates via a protocol called MAVLink [42] with the simulator (e.g., Gazebo [43]) to receive sensor data from the simulated world and send actuator command values to it. Furthermore, the PX4 SITL communicates with the GCS and an offboard API (e.g., ROS [44]) to send telemetry from the simulated environment and receive commands.

In the Gazebo simulator, the Yuneec Typhoon H480 hexacopter has been implemented (Figure 9) along with a series of sensors’ plugin, namely, the GNSS and the camera, which triggers at a frequency of 1 Hz and provides the pictures that are analyzed in real-time in the way presented in the previous section.

4.2. Field Case Study

A real PV plant configuration has been used as a real case study. The selected PV plant is located in Massa Lombarda, Ravenna, Italy. It is a 4.72 MWp PV plant with polycrystalline modules. The plant is connected to 4 inverters and then to the medium-voltage grid.

The digitalized map of this PV plant is represented in Figure 10. The plant is divided in several rows of different length, each row is inclined of 30 degrees with respect to the ground and the width comprises two modules.

The plant has been imported into the Gazebo simulator as presented in Figure 11.

For the inspection operations, a proper horizontal routing has been designed (Figure 12). This helps the postprocessing of the acquired information and it does not worsen the acquisition time.

The route is appropriately shifted with respect to the target panels to simulate the map georeferencing error the planning process is typically subjected to.

5. Flight Simulation Results

In Figure 13, in which the proposed correction object of this paper is not activated (this scenario represents the state-of-the-art of automatic flight), the UAV follows quite well the given route, even in presence of GNSS errors. However, in the case of georeferencing errors, like the one presented in the previous section, due to the UAV wrong positioning, the camera is not able to capture part of the PV row.

In particular, as can be observed in the presented plot, the last row before landing sees a complete slice of the PV row not covered. Thus, the data acquisition related to that plant section is completely ineffective.

Conversely, in Figure 14, the results related to the proposed correction are reported. It is worth noting that, in this case, the UAV platform is able to correct the pre-planned setpoints due to information coming from the captured images (the so called fly by sensor) and modifies its trajectory going towards the target PV row. In this way, the camera FOV is able to completely cover the desired target. The only exception to this positive corrective effect regards the transition between one row to another in which the correction mechanism requires some time to take place. To mitigate this effect, it is recommended to properly adjust UAV flight speed in those transition between one row to another.

6. Conclusions

In this paper, the problem of improving the performance of automatic inspection for PV plants has been considered and a vision-based guidance law has been developed in this scope.

The proposed technique exploits information from sensors that are already installed onboard, thus it does not require relevant additional equipment costs with respect to traditional UAV flights. In addition, the automatic inspection cost compared to the energy production of a more than 4 MW PV plant is fully payback in one year considering just a 1.5% rate of PV module failures.

A dynamic compensator, which takes as input the sensor camera information, has been employed through the combination of a Canny edge-detector and a Hough transform, giving as output the needed correction to the set-point position to reject errors with respect to the desired trajectory. To validate the overall system, resembling application on a real hardware, the PX4 firmware software-in-the-loop testing environment has been used. In particular the Gazebo modeling environment has been exploited, thanks to its capability to test sensors’ feed like visual cameras. The results show that the approach is successful in reducing the number of pictures with incomplete coverage of the target PV row if compared to the state of the art solutions. Note that the developed vision algorithm runs in real-time in simulation and that tests on a typical embedded computer of a UAV (NanoPi NEO Air: http://wiki.friendlyarm.com/wiki/index.php/NanoPi_NEO_Air) show a computational time of less than 50 ms on a 640 × 360 pixels image.

Possible limitations of the proposed approach regard the robustness of vision-based methods against changes in lighting and environmental conditions. Comparing the results with the state-of-the-art of visual servoing presented in the introduction, higher steady-state positioning errors can be highlighted. This is because, in direct visual servoing, the captured images are used in the system feedback to improve in real-time the robot accuracy. On the other hand, that kind of controller rates are bounded by the vision system frequency, possibly leading to stabilization problems. In the proposed solution, the system simply change the set-point position, whereas stabilization is guaranteed by the low-level controller even in the presence of faults of the vision system.

Image quality is a key factor in automated and semi-automated image processing techniques that are very often used in PV plant inspection. The results found in this paper shows that the proposed technique can improve drastically the image quality for autonomous flight. Moreover, with this technique the flight altitude can be reduced, and thus the resolution of the images can be higher with a fixed equipment.

This manuscript may represent a very important step toward the autonomous flight and UAV-based automatic monitoring operations in energy plants. In particular, for large-scale PV plants, where the cost-performance ratio is a crucial aspect and manned lights are difficult due to typical harsh environments, fly-by-sensor automatic flights allow higher accuracy and more frequent inspection giving a more updated plant status report.